SoLar: Sinkhorn Label Refinery for Imbalanced Partial-Label Learning
Abstract
Partial-label learning (PLL) is a peculiar weakly-supervised learning task where the training samples are generally associated with a set of candidate labels instead of single ground truth.
While a variety of label disambiguation methods have been proposed in this domain, they normally assume a class-balanced scenario that may not hold in many real-world applications.
Empirically, we observe degenerated performance of the prior methods when facing the combinatorial challenge from the long-tailed distribution and partial-labeling.
In this work, we first identify the major reasons that the prior work failed.
We subsequently propose
SoLar , a novel Optimal Transport-based framework that allows to refine the disambiguated labels towards matching the marginal class prior distribution.
SoLar additionally incorporates a new and systematic mechanism for estimating the longtailed class prior distribution under the PLL setup.
Through extensive experiments,
SoLar exhibits substantially superior results on standardized benchmarks compared to the previous state-of-the-art PLL methods.
Code and data are available at: https://github.com/hbzju/SoLar .
Introduction
The remarkable success of deep learning typically requires collecting massive labeled data, which is notoriously labor-intensive.
Of particular concern, data labeling can suffer from inherent and pervasive label ambiguity.
Take Figure 1 (a) as an example, a Mule can be visually similar to both
Donkeys and Horses, which hinders non-expert annotators from recognizing the true breed.
Similar cases also arise in many real-world applications, such as automatic image annotation [1], bird song recognition [2], and facial age estimation [3].
To deal with ambiguous supervision, partial-label learning (PLL) [4, 5], which allows each training example to be annotated by a set of candidate labels, has attracted significant attention from the community.
A plethora of methods have been developed to tackle this problem, including pseudo-labeling methods [6, 7], graph-based methods [8, 9, 10], etc.
Despite the promise, existing PLL methods have been commonly driven by the assumption that training data consists of class-balanced distribution, which may not hold in practice.
In many realworld scenarios, training data exhibits a long-tailed label distribution [11].
That is, many labels occur infrequently in the training data.
Concerningly, the imbalanced data can exacerbate the fundamental
[Figure]
challenge of label disambiguation, i.e. , identifying the ground-truth label from candidates.
Indeed, we find that current best-performing PLL methods display degenerated performance in the long-tailed setting.
This happens because the predictions of pseudo-labeling—a core component that PLL methods rely on—can be largely biased towards the head and majority classes.
We exemplify the phenomenon in Figure 1 (b), where a strong method PRODEN [6] rarely assigns probability mass to the tail classes (indexed by 9 and 10).
As a result, these tail classes suffer from almost zero accuracy, as shown in Figure 1 (c).
To date, few efforts have been made to resolve this.
Motivated by this, we propose a novel framework for long-tailed partial-label learning, called
S inkh o rn La bel r efinery (dubbed SoLar ).
Our framework emphasizes the long-tailed nature of training data, while performing label disambiguation for partial-label learning.
Our key idea is to enforce constraints on pseudo-labels—the distribution of which should match the class prior distribution .
The constraints can incentivize the label disambiguation process to select tail labels from candidate sets, instead of always selecting head classes.
We formalize the idea as an optimal transport problem [12] that searches for a proper label assignment subjected to constraints: (1) the probability mass is distributed within candidate sets; (2) the distribution of (predicted) pseudo-labels matches the prior class distribution.
The first constraint is natural in the PLL setup, and the latter one uniquely empowers SoLar to disambiguate tail labels from candidates.
We show that our constrained optimization objective can be tractably solved using the Sinkhorn-Knopp algorithm [13], which incurs only a few computational overheads.
Theoretically, we prove that our overall objective is statistically consistent, which ensures the optimality of the learned classifier at a population level.
As an integral part of our framework, SoLar tackles the challenge of estimating the class prior distribution, which is necessitated in our constrained optimization.
Unlike conventional long-tailed learning (LTL) scenarios [14, 15], the PLL setup is presented with label ambiguity in the proximity of candidate sets.
This makes it intractable to estimate the marginal class prior distribution, especially by counting training samples grouped by class labels.
We tackle this non-trivial problem by performing an iterative class prior estimation, which provides a strong proxy for real prior (Section 3.2).
Empirically in Figure 1 (b), we show that the estimated class prior favorably matches the ground-truth prior on a long-tailed version of the CIFAR10 dataset with partial labels.
We comprehensively evaluate SoLar on various benchmark datasets, where SoLar establishes stateof-the-art performance.
Compared to the best baseline, SoLar improves the accuracy on tail classes by 15.12 % on the long-tailed version of the CIFAR10 dataset.
While our work primarily focuses on label disambiguation from the PLL perspective, we demonstrate that SoLar can be compatible with
LTL techniques as well.
For example, we equip SoLar with logit adjustment [14], a state-of-the-art
LTL method and further improve SoLar’s performance by 6.72 % on long-tailed CIFAR10.
We hope our work will inspire future works to tackle this important problem.
Problem Setup
Let X =
R d denote the input space and Y = { 1 , . . .
, L } denote the label space.
We receive a training dataset D
= { ( x i , S i ) } ni =1 with n examples.
Each tuple in D consists of an image x i ∈ X and
// Compute pseudo
end
end
// Empirical class prior estimation
18 end the candidate label set S i ⊂ Y .
Following previous works [6, 7], we assume the true label y i
∈
Y of x i is concealed in S i , i.e. , y i ∈ S i .
A fundamental challenge in partial-label learning is label disambiguation, i.e. , identifying the ground-truth label y
i from the candidate label set S i .
Our goal is to train a classifier f : X (cid:55)→ [0 , 1] L , parameterized by θ , that can perform predictions on unseen testing data.
Here f is the softmax output of a neural network, and f j ( · ) denotes the j -th entry.
Besides, we denote the prediction matrix by P =
[ p 1 , . .
. , p n ] ⊤ =
[ p ij ] n × L , where p ij = f j ( x i ) .
To perform label disambiguation, we maintain a pseudo-label matrix Q =
[ q 1 , . . . , q n ] ⊤ ∈ [0 , 1] n × L and train the classifier with the cross-entropy loss l ce ( f ; x i , q i ) =
(cid:80)
Lj =1 − q ij log( f j ( x i )) , where q ij is the j -th entry of q i .
Proposed Method
In this section, we describe our novel S inkh o rn La bel r efinery (SoLar) framework for partial-label learning.
As we reckon, the combinatorial challenge of partial-labeling and long-tail learning lies in matching between a decent marginal prior distribution with drawing the pseudo labels.
To cope with these problems, SoLar comprises two components.
On one hand, SoLar formulates an optimal transport objective to facilitate the refinement of pseudo-labels to match the marginal class prior.
The resulted constrained optimization objective encourages the model to properly draw the tail labels.
On the other hand, the PLL setting manifests difficulty in straightforwardly estimating the class prior.
SoLar henceforth incorporates two novel techniques—a moving-average estimation approach with a complementary reliable set sampling mechanism.
The pseudo-code is summarized in Algorithm 1 .
3.1 Sinkhorn Pseudo-Label Refinery
Optimal Transport Objective.
We formalize an optimal transport problem for proper label assignments.
At each training step, SoLar searches for pseudo-labels Q close to the current classifier’s predictions P , while subject to some constraints:
[Equation]
r is an L -dimensional probability simplex that indicates the prior class distribution.
In other words, we would like the summation of the j -th column in Q ( i.e. , the total probability mass predicted as class j ) to match the prior class probability.
Note that, here we temporarily assume we have a decent estimation of the class priors and we will describe the means of estimation in the following Section 3.2.
The column vector c = 1 n 1 n indicates that our n training examples are sampled uniformly.
In Eq.
(1), with a slight abuse of notation, Q is essentially a joint probability matrix post to the refinery.
That is, each x i is treated as a supplier holding 1 n units of goods while the j -th label needs q ij units of goods.
To obtain pseudo-labels, we take a re-scaled solution n Q bearing an assumption of a uniform marginal 1 n of x i .
With further scrutiny of the objective function above, this formulation complies with two constraints we posited, such that: (i)-it avoids biased assignment towards (only) the head labels in a long-tailed
PLL setup, with Q ⊤ 1 n = r constraining the distribution of pseudo-labeling to match the prior of the class labels.
(ii)-the probability mass of Q in each row is driven to be distributed within the candidate label set of a sample x i .
In Eq.
(1), we use − log( p ij ) as cost such that predictions with higher confidence enjoy a lower cost.
This can also be replaced by other cost functions h ( p ij ) as long as h is monotonically decreasing.
But, we show that the negative log-likelihood cost results in a statistically consistent risk estimator.
Theorem 1.
Denote ˜∆ the set of measures on X × Y that satisfies the constraints in Eq.
Let D p be the distribution of tuple ( x , S ) .
The population risk of our method is as follows,
[Equation]
Let f ∗∗ and f ∗ denote the optimal classifiers trained on fully-supervised data with cross-entropy loss and PLL data with R ( f ) , respectively.
Assume the small ambiguity condition [5] is satisfied.
Then, under the deterministic scenario and ignoring the null set, we have f ∗ =
f ∗∗ .
We refer the readers to Appendix A.1 for the proof.
Theorem 1 indicates that the optimal classifier f ∗∗ can be recovered by our algorithm at a population level.
Compared with other theoretically consistent
PLL risks [6, 16], our risk preserves the marginal class distribution of the estimated pseudo-labels, and is reasonably more favorable in the long-tailed learning setup.
Fast Approximation.
To resolve Eq.
(1), we adapt the well-known Sinkhorn-Knopp algorithm [13] for efficient optimization.
Formally, we define a matrix M such that m ij =
p λij I ( j ∈ S i ) , where λ > 0 is a smoothing regularization coefficient.
The pseudo-labels are obtained by,
[Equation]
with iteratively updated α ← c .
[Equation] [Equation]
Here ./ denotes element-wise division.
α ∈ R n and β ∈ R L are known as scaling coefficients vectors.
Detailed derivation are provided in Appendix A.2.
Here, we set m ij =
if j / ∈ S i , which is equivalent to define log(0) = − inf .
Thus, Q will also be assigned
s outside the candidate sets.
Inspired by [17], we also involve a queue acceleration trick to avoid traveling the whole training set.
We found that Eq.
(3) typically converges to a satisfactory result in < 50 rounds.
With only simple algebra operations, this can be efficiently implemented on GPUs; see Appendix C.3.
3.2 Iterative Class Prior Estimation
Unlike conventional long-tailed learning scenarios [14, 15], the PLL setup is presented with label ambiguity in the proximity of candidate sets.
This makes it intractable to estimate the marginal class prior distribution, especially by counting training samples grouped by class labels.
In SoLar, towards estimating the class marginals, our aim is to propose a solution with (almost) full autonomy that can adapt to the data distribution and the training state of the model.
Here, we propose two interdependent techniques to achieve this goal.
The resulted estimation is utilized in Eq (1).
The core of our method is two-fold: (i)-we propose a moving-average style updating rule for class prior estimation, resulting in stable training dynamics ; (ii)-we carefully devise a sample selection mechanism for robust training, while preserving the marginal label distribution as much as possible.
Moving-Average Distribution Updating.
We propose using the model predicted results as a proxy for class prior estimation.
In spite of its simplicity, we cannot ignore the fact during the early stage of training, the predicted labels can be largely imprecise, not trustworthy, and biased towards the head labels.
Therefore, we propose an update mechanism in a moving-average fashion.
Since we hold no assumption on the true class prior, the updating scheme is initiated from a uniform class prior r =
[1 /L, . . .
, 1 /L ] .
Respectively at each cycle (often defined by epochs), SoLar refines the distribution as follows.
[Equation]
where µ ∈ [0 , 1] is a preset scalar.
As displayed, the class prior is progressively updated alongside stabler training dynamics through a linear function.
As the training proceeds, the model becomes more accurate and thus the estimated distribution becomes more reliable.
Class-wise Reliable Sample Selection.
In the early stages of training, the class prior estimation can be inaccurate due to the poor pseudo label predictions.
Thus, we posit that the lack of proper sample selection at early stages may jeopardize the marginal prior estimation.
As common as it is for
PLL learning dynamics, we want to reduce the variance of the pseudo-label quality, via a reliable sampling algorithm.
This module works together with the moving-average class prior estimation.
A brute-force but effective strategy is to employ the small-loss sampling criterion in noisy-label learning [18], which assumes low-quality pseudo-labels can likely be characterized by a large loss.
However, in a long-tailed setup, the tail labels can also incur a large loss because of their low-shot nature, making them rarely selected.
We empirically show this phenomenon in Section 4.3.
We hence implement the following procedures into the proposed SoLar framework.
The goal of such a module is to make sure the coverage of the tail labels as well as to keep the marginal prior unbiased.
To begin with, to make sure that all labels — particularly the tail labels — are coverd, we split the batch of data B into L slices that are indexed by the argmax taken from the refined pseudo-labels:
B j = { ( x i , q i ) | j = arg max j ′ ∈ S
i q ij ′ } ⊂ B .
In what follows, we derive two combinatorial rules on the selectivity of the training samples:
(i)
Based on the aforementioned setup, we modify the small-loss criterion in the noisy-label learning [18] and adapt it to a long-tailed setup.
Specifically, we first compute the instance-wise loss by marginalizing over the class index: l i = − (cid:80)
Lj =
q ij log( p ij ) .
Then, within the provided set B j , we anchor the total sample number to be k = min( ⌈ r j ρ | B |⌉ , | B j | ) , which is associated with r j for distribution-preserving .
The samples are being selected with top- k smallest l i .
(ii)
A parallel mechanism is inspired by the application of pseudo-labeling schemes in semisupervised learning [19].
To adopt such approaches in SoLar, we purposely select those samples having a very high confidence score which is measured by the cosine similarity: e i = q ⊤ i p i .
As such, this second criteria selects samples that meet the inequality e i > τ .
This mechanism further ensures high utility on the (likely) well-disambiguated samples.
Notice that, ρ, τ > 0 are two thresholding hyper-parameters.
The two introduced mechanisms are established to work collaboratively.
Simply put, samples that satisfy either criterion are used for training the classifier.
We will show later in Section 4.3 the efficacy of our sample selection mechanisms.
Experiments
In this section, we experimentally analyze the proposed SoLar method under various scenarios for the imbalanced PLL problem.
More empirical results can be found in Appendix C.
4.1 Setup
Datasets.
First, we evaluate SoLar on two long-tailed datasets CIFAR10-LT and CIFAR100-LT introduced in [20, 21].
The training images are randomly removed class-wise to follow a pre-defined imbalance ratio γ = n 1 n L , where n j is the image number of the j -th class.
For convenience, class indices are sorted based on the class-wise sample size, in descending order with n 1 ≥ ... ≥ n L .
We have n 1 = 5 , 000 for CIFAR10-LT and n 1 = 500 for CIFAR100-LT.
We use different imbalance ratios to evaluate the performance of SoLar, with γ ∈ { 50 , 100 , 200 } for CIFAR10-LT and γ ∈ { 10 , 20 , 50 } for CIFAR100-LT.
We then generate partially labeled datasets by manually flipping negative labels ¯ y ̸
= y to false-positive labels with probability ϕ =
P (¯ y ∈ Y | ¯ y ̸ = y ) , which follows
[Table]
the settings in previous works [
[Table]
The final candidate label set is composed of the groundtruth label and the flipped false-positive labels.
We choose ϕ ∈ { 0 . 3 , 0 . 5 } for CIFAR10-LT and ϕ ∈ { 0 . 05 , 0 . 1 } for CIFAR100-LT.
For all experiments, we report the mean and standard deviation based on 3 independent runs (with different random seeds).
Baselines.
We compare SoLar with six state-of-the-art
partial-label learning methods: 1) PiCO [7] leverages contrastive learning to disambiguate the candidate labels by updating the pseudo-labels with contrastive prototype labels.
2) PRODEN [6] is also a pseudo-labeling method that iteratively updates the latent label distribution by re-normalized classifier prediction.
3) VALEN [23] assumes the candidate labels are instance-dependent and recovers the latent label distributions by a Bayesian parametrization model.
4) LWS [22] also works in a pseudo-labeling style, which weights the risk function by considering the trade-off between losses on candidate labels and non-candidate ones.
CC [16] is a classifier-consistent method that assumes the candidate label set is uniformly sampled.
6) MSE and EXP [24] utilize mean square error and exponential loss as the risk estimators.
All the hyper-parameters are searched according to the original papers.
Implementation details.
We use an 18-layer ResNet as the feature backbone.
The model is trained for 1000 epochs using a standard SGD optimizer with a momentum of 0 .
The initial learning rate is set as 0 . 01 , and decays by the cosine learning rate schedule.
The batch size is 256 .
These configurations are applied for SoLar and all baselines for fair comparisons.
For SoLar, we devise a pre-estimation training stage, where we run a model for 100 / 20 epochs (on CIFAR10/100-LT) respectively to obtain a coarse-grained class prior.
After that, we re-initialize the model weights and run with this class prior.
For our Sinkhorn-Knopp algorithm, we fix the smoothing regularization parameter as λ = 3 and the length of the queue for acceleration as 64 times batch size.
The movingaverage parameter µ for class prior estimation is set as 0 .
in the first stage and fixed as 0 .later.
For class-wise reliable sample selection, we linearly ramp up ρ from 0 .
2 to 0 .
6 in the first 50 epochs and fix the high-confidence selection threshold τ as 0 . 99 .
To improve the representation ability of SoLar, we further involve consistency regularization [19] and Mixup [25] techniques on reliable examples; see Appendix B.1 for more details.
For fair comparisons, we equip all the baselines except PiCO with these two techniques as well.
[Table] [Table]
4.2 Main Results
SoLar achieves SOTA results.
As shown in Table 1, SoLar significantly outperforms all the rivals by a notable margin under various settings of imbalance ratio γ and label ambiguity degree ϕ .
Specifically, on CIFAR10-LT dataset with ϕ = 0 .
5 , we improve upon the best baseline by 8.29 %, 9.28 %, and 7.47 % when the imbalance ratio γ is set to 50 , 100 , and 200 respectively.
Moreover, most baselines display significant performance degradation as ϕ increases, whereas SoLar remains competitive.
We also verify the effectiveness of SoLar on a more challenging CIFAR100-LT dataset with 10 times more classes and thus stronger label ambiguity.
Notably, the performance gap between
SoLar and baselines remains substantial.
An interesting observation is that CC obtains impressive performance on the CIFAR-100 dataset, but is still inferior to SoLar in most cases.
These observations clearly validate the superiority of SoLar.
Results on different groups of labels.
We show that SoLar achieves overall strong performance on both head and tail classes.
To see this, we report accuracy on three groups of classes with different sample sizes.
Recall from Section 4.1 that the class indices are sorted based on the sample size, in descending order.
We divide the classes into three groups: many ( { 1 , 2 , 3 } ), medium ( { 4 , 5 , 6 , 7 } ), and few ( { 8 , 9 , 10 } ) shots for CIFAR10-LT and many ( { 1 , . .
, 33 } ), medium ( { 34 , . .
. , 67 } ), and few ( { 68 , . .
, 100 } ) shots for CIFAR100-LT.
Table 2 shows the accuracy of different groups on both
CIFAR10-LT and CIFAR100-LT.
On few-shot classes, the gaps between SoLar and the best baseline are 15.12 % and 2.57 % on CIFAR10-LT and CIFAR100-LT.
Moreover, SoLar retains a competitive performance on many-shot labels ( i.e. , head classes).
In contrast, most baselines exhibit degenerated performance on tail classes, especially on the CIFAR100-LT dataset.
The comparisons highlight that
SoLar poses a good disambiguation ability in the long-tailed PLL setup.
4.3 Ablation Studies
In this section, we present our main ablation analysis to show the effectiveness of SoLar.
The main results of our ablation studies are shown in Table 3; more results can be found in Appendix C.4.
[Figure]
Effect of Sinkhorn label refinery.
It is of interest to see whether the performance improvement comes from our Sinkhorn label refinery procedure.
To see this, we equip PRODEN with our sample selection and class prior estimation mechanism (dubbed PRODEN w S+CPE ).
As shown in Table 3,
PRODEN w S+CPE still fails to disambiguate labels on the tail, which validates the superiority of our Sinkhorn label refinery procedure.
Effect of class prior estimation.
To verify the effectiveness of class prior estimation in SoLar, we compare with three variants: 1) SoLar w/o Pre-Est which removes the pre-estimation stage of SoLar and trains from uniform class prior; 2) SoLar w Cand-Est initializes the class prior distribution by counting per-class candidate numbers, i.e. , r j = (cid:80) ni
=
I ( j ∈ S i ) ; 3) SoLar-Oracle also removes the pre-estimation stage but trains from the real class prior.
From Table 3, we draw three salient observations.
First, SoLar outperforms SoLar w/o Pre-Est, which suggests that using a coarselyestimated class prior is more beneficial than using a uniform one.
Secondly, SoLar w Cand-Est obtains competitive results, showing an alternative way for initializing r .
Finally, we contrast with
SoLar-Oracle, which theoretically serves as an upper bound by using the real class prior.
We observe that SoLar’s performance favorably matches or even outperforms SoLar-Oracle on few-shot classes.
The results are encouraging since SoLar does not require access to real class prior at all.
Effect of sample selection.
Next, we experiment with different sample selection strategies, including:
) SoLar
w/o S which regards all examples as clean samples and does not perform selection; 2) SoLar w GS which performs selection in a global manner rather than class-wise; 3) SoLar w/o S-SL which only selects high-confidence examples.
From Table 3, we can observe that SoLar w/o S worsened the performance because many pseudo-labels are unreliable at the beginning.
SoLar w/o GS works relatively well on data-rich labels but suffers on the tail.
This happens because the majority classes dominate the sample selection.
SoLar w/o S-SL performs similarly to SoLar w/o GS on CIFAR10-LT, but diverges on CIFAR100-LT.
In the beginning, few examples are selected as the classifier is less confident, resulting in degenerated solution.
Our results indicate that our sample selection procedure is indispensable to SoLar in handling long-tailed class distribution.
4.4 Further Analysis
Combining SoLar with logit adjustment.
While our work mainly focuses on label disambiguation from the PLL perspective, potential improvement is straightforward by applying long-tailed learning techniques to well-disambiguated datasets.
To this end, we equip SoLar with logit adjustment [14] (dubbed SoLar-LA ), a state-of-the-art LTL technique, for improved performance.
Denote the logits of our model by g ( · ) , i.e. , the output of the network prior to the softmax layer.
Given a testing data x , SoLar-LA made prediction by arg max j ∈
[ L ] g j ( x ) − ζ · log( r j ) , where ζ is a temperature parameter for calibration.
This post-hoc corrected prediction is known to be theoretically consistent for minimizing the balanced error [14].
Note that logit adjustment cannot be applied to the original
PLL data directly, as it requires the true class prior as well as a classifier trained by minimizing the long-tailed empirical risk.
Fortunately, SoLar achieves both goals.
Hence, we implement SoLarLA by using the classifier as well as the estimated class prior of SoLar.
Figure 2 (a) reports the performance of SoLar-LA with varying ζ values.
With a proper ζ , SoLar-LA improves upon SoLar by 6.72 % on CIFAR10-LT.
This suggests that SoLar is compatible with existing LTL methods and opens the door to exploring more advanced LTL techniques.
Results on fine-grained partial-label learning.
In practice, semantically similar classes can lead to significant label ambiguity, as exemplified in Figure 1 (a).
To test the limit of SoLar, we follow PiCO [7] and evaluate on two fine-grained datasets: 1) CUB200-LT [26] dataset with 200 bird species; 2)
CIFAR100-LT with hierarchical labels (CIFAR100-H-LT), where the candidate labels are generated within the same superclass 2 .
We set ϕ = 0 .
05 , γ = 5 for CUB200-LT and ϕ = 0 .
5 , γ = 20 for
CIFAR100-H-LT.
In Figure 2 (b) and (c), we compare SoLar with two strongest baselines PRODEN and PiCO, where SoLar improves the best baselines by a large margin ( 4.37 % on CUB200-LT and 3.67 % on CIFAR100-H-LT).
These results clearly validate the effectiveness of SoLar, even when the dataset presents severe label ambiguity.
[Table]
Results on real-world long-tailed learning data.
It is of interest to verify the effectiveness of SoLar on real-world imbalanced datasets.
To this end, we conduct experiments on the large-scale SUN397 dataset [27] containing 108,754 RGB images and 397 scene classes; more results on real-world and imbalanced
PLL datasets are shown in Appendix C.6.
For the
SUN397 dataset, we hold out 50 samples per class for testing and the resulting training set has an imbalanced ratio of 2311 / 50 ≈ 46 .
Similarly, we synthesize a partial-label dataset set with an ambiguity degree of ϕ = 0 .
05 ; see Appendix C.1 for more details.
From Table 4, we can observe that SoLar still outperforms the baselines by a substantial margin.
Related Work
Partial-label Learning (PLL) [4] allows each example to be equipped with a candidate label set with the ground-truth label being concealed.
A plethora of PLL methods [5, 28, 29, 30, 31] have been developed.
For example, maximum margin-based methods [32, 33] maximize the margins between candidate labels and the remaining ones.
Graph-based approaches [34, 8, 9, 10, 35] typically leverage nearest neighbors to smooth the logical label vectors.
Recently, pseudo-labeling methods [16, 6, 22, 36, 7, 37] become increasingly prevalent due to their promising performance, e.g. , PRODEN [6] re-normalizes classifier’s outputs and PiCO [7] updates pseudo-targets by using contrastively learned prototypes.
However, most of these works overlook the fact that class distribution can be imbalanced.
Our empirical studies demonstrate that state-of-the-art PLL methods fail to disambiguate true labels on the minority classes, which motivates us to explore the optimal transport-based label disambiguation technique for PLL.
Long-tailed Learning (LTL) assumes that the training data follow Zipf’s law that few labels pose a large number of instances, which is observed in a wide range of applications [11].
To cope with the LTL problem, sampling-based methods [38, 39] re-balance different classes at a data level, e.g. , up-sampling on minority classes.
Re-weighting methods [20, 14, 40, 41, 15] typically adjust the weights of instances or classes that calibrates the classifier predictions to be balanced.
Some works also explore transfer learning [42, 43] or ensemble learning [44, 45] in LTL.
A recent trend in LTL [46, 47] is to decompose representation and classifier learning.
Nevertheless, these methods assume a full-supervision of training data, which is unrealistic in many real-world applications.
It has motivated researchers to study the LTL problem in weak-supervision scenarios, such as semi-supervised learning [48, 21] and noisy-label learning [49, 50, 51].
In this work, we explore a new formulation of LTL where only candidate labels are provided.
Optimal Transport (OT) [12] is originally proposed to depict the distance between two measures.
Recently, it has drawn huge attention from different fields in machine learning, including semisupervised learning [52], domain adaptation [53, 54], object detection [55], and generative models [56].
The most related one to our work is OTLM [15], which also applies OT to the supervised LTL problem.
By assuming a known test class distribution, OTLM performs OT-based post-hoc correction on model predictions to obtain balanced results.
In contrast, our work targets the long-tailed PLL problem and holds no assumption on class priors.
Conclusion
In this work, we present a novel Sinkhorn label refinery framework (SoLar) for a challenging imbalanced partial-label learning problem.
Key to our method, we derive an optimal transport objective that refines the pseudo-labels to match the true class prior.
Additionally, we tackle the non-trivial class prior estimation problem by a moving-average technique, which is further guarded by a sample selection mechanism for controllable model training.
Empirically, SoLar can run without knowledge of the real class distribution.
Comprehensive experiments show that SoLar improves baseline algorithms by a significant margin.
As we collect the real-world applications, the partiallabeled tasks often arise together with a long-tailed class distribution.
That is, from the perspective of human annotators, a long-tailed label is much harder to tag and that may subsequently lead to a partial-labeled candidate set.
In the future, we hope to explore this line of research in broader tasks.
We hope our work will draw more attention from the community towards a broader view of tackling the imbalanced partial-label learning problem.
Acknowledgments and Disclosure of Funding
This work was largely supported by the Key R&D Program of Zhejiang Province (Grant No.
2020C01024).
JZ was supported by the Fundamental Research Funds for the Central Universities (No. 226-2022-00028).
HW, MX were partially supported by the Key Research and Development
Program of Zhejiang Province of China (No. 2021C01009) and Fundamental Research Funds for the Central Universities.
LF was supported by the National Natural Science Foundation of China (Grant
No. 62106028), Chongqing Overseas Chinese Entrepreneurship and Innovation Support Program.
Appendix A Theoretical Proofs A.1 Proof of Theorem
First, we provide the following lemma to show the consistency of the standard cross-entropy loss.
Lemma 1.
If the cross-entropy loss is used as loss function, the optimal classifier f ∗∗ that minimizes the population risk R CE ( f ) =
E x ,y [ − log( f y ( x ))] satisfies f ∗∗ y ( x )
=
p ( y = i | x ) .
Proof.
We provide a proof sketch in the sequel, and a similar result has been shown in [57, 16].
The cross-entropy loss leads to the following optimization problem,
[Equation]
By introducing a multiplier ξ , the corresponding Lagrangian is as follows,
[Equation]
Setting the derivative to 0 yields that,
[Equation]
We conclude that ξ = 1 and f ∗∗ y ( x ) = p ( y = i | x ) holds.
Next, we provide the definition of the (theoretical) ambiguity degree [5] to establish the learnability of the PLL problem.
Definition 1.
Denote the distribution of triplet ( x , y, S ) by ˆ D p , We define the ambiguity degree as,
[Equation]
We say a data distribution satisfies the small ambiguity degree condition if κ < 1 .
This is a natural requirement.
If κ = 1 , there exists at least one label pair that always co-occur, and thus, it is impossible to find the optimal hypothesis given partial-labeled data.
Now, we provide the main proof sketch for Theorem 1.
Note that we always seek an optimal joint probability matrix before model training, which is mainly designed for empirical measures of the data samples.
At a population level, we aim to search for an optimal probability measure that meets the marginal constraints and candidate constraints.
Note that we aim to minimize
E ( Q , P ) =
KL ( Q || P ) + H ( Q ) where KL ( · ) is the Kullback–Leibler divergence and H ( · ) is an entropy regularizer.
This gives rise to our population risk,
[Equation]
Here we omit the entropy term H ( ϖ ) as it serves as a regularizer.
As the training labels are categorical, we may assume H ( p ( y | x ))
= 0 , and thus, the infimum still holds.
Otherwise, we can offset it by setting the smoothing parameter λ = 1 in the Sinkhorn-Knopp approximation.
When the pseudo-labels are fixed, this objective is exactly the cross-entropy loss whose target measure is ϖ .
To show the consistency, we first prove that f ∗∗ is the minimizer of R ( f ) .
Substituting f ∗∗ into
R ( f ) and combining with Lemma 1, we obtain that,
[Equation]
It is obvious that ϖ = p ( y | x ) leads to the minimal KL-divergence.
On the other hand, we show that f ∗∗ is the unique solution.
We assume there exists another hypothesis f ′ that minimizes Eq.
(9) and holds a different prediction
y ′ ̸
= y from f on at least one instance ( x , y, S ) .
By the deterministic assumption, we have R ( f ′ )
= 0 , which holds only if y ′ is invariably included in the candidate set S , i.e. , P ˆ D p ( y ′ ∈ S
| x , y ) = 1 .
Clearly, it violates the small ambiguity degree condition and causes a contradiction.
Ignoring all the null set where p ( x , y ) =
0 , we conclude that the minimizer f ∗ of our population risk equals the fully-supervised f ∗∗ .
The above discussion indicates that our method poses good performance guarantees like existing
PLL methods [6, 16].
A similar loss R ( f ) =
E ( x ,S )
∼D p min y ∈ S − log( f y ( x )) is also discussed in [6], which is known as minimal loss .
Its main difference from our objective is that we seek the best probability measure instead of an example-wise minimum.
Our risk preserves the marginal constraint of the estimated pseudo-labels and is reasonably more favorable in the long-tailed learning setup.
A.2 Derivation of the Sinkhorn-Knopp Iteration
In this section, we briefly introduce the derivation of the Sinkhorn-Knopp algorithm to ensure the integrity of our work.
Recall that Eq.
(1) is a standard linear programming (LP) problem, and can be solved in polynomial time.
But, considering the high volume of data points as well as potential large class space, common
LP solvers typically become time-consuming.
To this end, we investigate a smoothed version of this optimization problem for fast approximation.
Technically, we further add a negative entropy regularization term to obtain the following objective,
[Equation]
This entropy regularizer is derived from an optimization perspective and is different from the one in our derivation of E ( Q , P ) =
KL ( Q || P ) + H ( Q ) .
The resulting objective becomes smoothing and convex, and thus, can be efficiently solved.
It is worth pointing out that in the PLL problem, we have a special constraint that no probability mass is assigned outside the candidate label sets.
To avoid introducing another set of annoying Lagrange multipliers, we transform this constraint to infinity costs on non-candidate labels.
That is, we define a cost matrix T such that T ij = − log( p ij ) I ( j ∈ S i )
+
inf
I ( j / ∈ S i ) and here we assume 0 · inf = 0 .
Now, we obtain the Lagrangian of the new optimization problem,
[Equation]
where u , v are Lagrange multipliers.
As the original optimization is convex, the solution has to satisfy the Karush-Khun-Tucker (KKT) conditions.
Therefore, we have the following equations,
[Equation]
Let M be a matrix such that m ij =
e − λT ij = p λ ij I ( j ∈ S i ) .
We can get that,
[Equation]
Thereby, solving the primal problem in Eq.
(11) equals to finding the multipliers u and v .
Again, this is equivalent to get another two vectors α ∈ R n , β ∈ R L such that α i =
e − 12 − λu i and β j = e − 12 − λv j , which are also known as scaling coefficients vectors.
Then, we can get the following equation,
[Equation] [Equation]
Recall that we enforce Q to meet the following constraints,
[Equation]
This gives rise to an alternative coordinate descent algorithm for updating the scaling coefficients,
[Equation]
) , β ← r ./ ( M ⊤ α ) .
It is also known as the Sinkhorn-Knopp fixed point iteration.
We refer the readers to [58] for its convergence properties.
Empirically, we find that setting λ = 3 and running 50 steps are enough to get a satisfactory solution.
The final step is to take a re-scaled the minimizer n Q in Eq.
(15), since Q serves as a joint probability matrix and the posterior is calculated by p ( x ,y ) p
( x ) = np ( x , y ) .
Without loss of generality, we slightly abuse the notation Q to denote the obtained pseudo-labels.
The core of our algorithm is to replace our zero constraints on non-candidates with infinity cost, which equals defining log(0)
= − inf .
These infinity values are then mapped back to 0 s in the
Sinkhorn-Knopp iteration, ensuring the feasibility of the calculation.
B Practical Implementation
In this section, we describe several details of the practical implementation of SoLar.
B.1 Details of Representation Enhancement
A recent work PiCO [7] has shown that existing PLL methods are typically trapped in a disambiguation-representation dilemma , where the low-quality representation and imperfect label disambiguation mutually deteriorate each other.
It can be even worse in the imbalanced setup because of low-shot examples on the tail.
While PiCO pioneers the contrastive learning technique in
PLL for superior performance, we show this can be achieved by a much simpler design.
In contrast to the complicated network architecture in PiCO [7], we involve consistency training regularizer along with Mixup augmentation to improve the representation quality.
Consistency Regularization.
Recently, consistency regularization (CR) has been widely applied in weakly-supervised learning [19], which assumes that a classifier should produce similar class probability for a sample and its local augmented copies.
Motivated by this, we also incorporated
CR into PLL.
Given an image x i , we adopt two data augmentation modules SimAugment [59] and
RandAugment [60] to obtain a weakly augmented image x wi and strongly augmented counterpart x si .
During training, the weak one is utilized to produce pseudo-labels by our Sinkhorn label refinery procedure as well as selecting the reliable subset.
Then, we define the the CR loss l cr = − (cid:80)
Lj =
q ij log( f j ( x si ))
that is the cross-entropy loss on x si and q i .
Mixup.
We further incorporate mixup training for improved performance.
Given a pair of weaklyaugmented examples x wi and x wj in the reliable set, we create a virtual training example by linearly interpolating both,
[Equation]
where σ ∼ Beta ( ς, ς ) and we simply set ς = 4 without further tuning.
Similarly, we define the mixup loss l mix as the cross-entropy loss on x m and q m .
In our implementation, we add CR and Mixup on reliable examples only.
For the remaining examples, since their pseudo-labels are unreliable, we train them with the classical re-normalized PLL loss l rn [6].
The final loss is defined by,
[Equation]
where l ce is the original classification loss of weakly-augmented instances.
We linearly ramp up η from 0 to 0 .
9 in the first 50 epochs, which helps warm up the classifiers.
B.2 Relaxed Solution for the Sinkhorn-Knopp Algorithm
Recall that we start from a uniform class prior.
Empirically, most of our experiments run very well.
But, in some cases, it can result in an unsolvable optimal transport objective.
For example, if there are only a few examples holding one specific label as a candidate ( ≪ n L ) , we have no hope to constrain the sum of pseudo-labels on this label to be the average number of instances.
Thus, the
Sinkhorn-Knopp iteration diverges.
In our implementation, once divergence occurs, we return a relaxed solution to the optimal transport problem.
Concretely, we modify the matrix M by adding a small number ϵ
= 1 e − 5 on its zero
[Table] [Table]
entries.
In other words, we regard all negative labels as potential candidates but assign them very large costs.
Now, M is an element-wise positive matrix
[Table]
and the Sinkhorn fixed iteration process guarantees to converge.
We can re-run the Sinkhorn-Knopp iteration to obtain a relaxed solution.
Finally
[Table]
we set the non-candidate labels as zero again, as they should not be assigned any probability mass.
By then, our algorithm can safely run with a uniform distribution.
This procedure is easy to implement and increases only 1 × computation at most.
Moreover, as the class prior is estimated better and better, we typically do not need this relaxed solution anymore.
In practice, it is not likely that we are fully unknowledgeable of the class distribution.
Thus, we can also use a good initialization to avoid the aforesaid problem.
Empirically, we find that initializing with the ratio of candidate label number is also a good choice for SoLar.
C Additional Experimental Results
In this section, we report the additional empirical results of our proposed SoLar framework.
All experiments are conducted on a workstation with 8 NVIDIA A6000 GPUs.
The licenses of our employed datasets are unknown (non-commercial).
C.1 Experimental Setups on Fine-grained Datasets and SUN
In the sequel, we show the full experimental setups on fine-grained classification datasets.
In particular, on CUB200-LT, we set the batch size as 128, and the length of the queue for Sinkhorn acceleration as 8 times batch size.
We train the model for 500 epochs without the pre-estimation training stage.
The ratio parameter ρ ramps up from 0.2 to 0.5 in the first 50 epochs.
Other hyper-parameters are the same as our default setting.
On CIFAR100-H-LT, we simply adopt the default parameter configurations.
The baselines are also fine-tuned to achieve their best results.
For the SUN397 dataset, we set the batch size as 128, and the queue length for Sinkhorn acceleration as 16 times batch size.
We train the model for 20/200 epochs for distribution estimation and regular training.
The gamma value is set as 0 .
1 at the pre-estimation stage.
Other parameters are the same as the CUB200-LT dataset.
As the SUN397 has a much larger scale, we calculate the empirical label distribution z by recording batch-wise statistics during training.
We find this on-the-fly counting strategy works as well as the default setup but is much faster.
As reported in Table 5, SoLar retains substantial performance advantages in different ambiguity degrees.
C.2 Full Results of SoLar with Logit Adjustment
In Table 6, we report the full results of SoLar-LA with varying ζ values on CIFAR10-LT and
CIFAR100-LT.
On CIFAR100-LT, SoLar-LA still outperforms SoLar with a proper ζ .
The results demonstrate that SoLar achieves promising disambiguation ability in the imbalanced PLL setup.
[Figure]
Given well-disambiguated data, SoLar makes it possible to apply off-the-shelf LTL methods for further improvements.
C.3 Running Time of the Sinkhorn-Knopp Algorithm
[Table]
As we mentioned in Section 3.1, our SinkhornKnopp algorithm can be efficiently implemented on GPUs.
In Table 7, we report the total running time ( 1000 epochs) of the Sinkhorn-Knopp iterations as well as model training based on our implementation.
We run SoLar with our default parameter configurations and evaluate using one
NVIDIA A6000 GPU.
During training, we maintain a queue of size 64 × 256 to store classifier predictions in previous 64 steps.
Then, we concatenate the prediction in the current batch with the queue to run the Sinkhorn-Knopp algorithm.
It can be shown that the Sinkhorn-Knopp iterations take less than 1 / 10 time cost than regular model training (including forward pass and backward propagation).
These results clearly validate the efficiency of our algorithm.
C.4 More Ablation
Effect of selection threshold ρ and τ .
We further investigate the effect of the small-loss selection ratio parameter ρ and the high-confidence threshold τ .
Figure 3 (a) shows the performance of
SoLar with varying ρ on CIFAR10-LT (without ramp-up).
When ρ = 0 , SoLar simply selects high-confidence samples, which leads to an unsatisfactory performance on few-shot labels.
The performance becomes much better as ρ becomes larger and achieves the best when ρ = 0 .
But, when ρ becomes too large, the model tends to overfit unreliable labels.
Empirically, we find that
SoLar works well in a wide range of ρ and ρ ≈ 0 .
5 is a good choice.
[Table]
Table 8 lists the result of SoLar with varying τ values.
With a relatively small τ , SoLar collects too many unconfident examples, which typically hurts the robustness of model training.
When τ = 0 .
99 , SoLar achieves rather promising results on different groups of labels.
A surprising observation is SoLar without high-confidence selection, i.e. , τ = 1 , achieves better results than SoLar on few-shot labels.
We find the reason is that those well-disambiguated examples on the majority classes are overlooked, which poses an effect of down-sampling and benefits learning on the minority classes.
As a negative effect, this sacrifices the performance of many-shot labels, which leads to an interesting trade-off.
We empirically set τ = 0 .
99 to retain relatively good performance on many-shot labels.
Effect of Sinkhorn smooth parameter λ .
Next, we study the effect of the Sinkhorn smooth parameter λ .
The results are shown in Figure 3 (b).
In general, a too small λ results in poor label
[Table]
assignment as the optimal transport objective becomes hard to be resolved.
With a too large λ , our objective is over-smoothed and thus the resultant solution may deviate from the true one
[Table]
which slightly degrades the performance as well.
We empirically found that λ
= 3 is a proper choice
[Table] [Table]
The role of unreliable examples.
In our implementation, we enable unreliable examples to be trained with re-normalized PLL loss since their pseudo-labels can be noisy.
This not only improves data utility but also serves as a warm-up mechanism for SoLar as we aim to train SoLar without knowledge of the true class prior.
To see the role of unreliable examples, we also evaluate SoLar without l rn and the results are shown in Table 9.
We observe that SoLar w/o l rn obtains favorable results on tail labels, even trained with only reliable examples.
This verifies the importance of our distribution-preserving sample selection mechanism.
Given the oracle class prior, SoLar w/o l rn obtains competitive performance and matches our main results.
In practice, proper utilization of unreliable samples typically leads to more promising results.
Ablation on representation learning.
Here we ablate the contributions of two components in representation enhancement: mixup augmentation training and consistency regularization.
Specifically, we compare SoLar with three variants: 1) SoLar w/o MU which removes Mixup augmentation training; 2) SoLar w/o CR which removes consistency regularization; 3) SoLar w/o MU+CR which removes both Mixup and consistency parts; 4) PRODEN w/o MU+CR is the PRODEN algorithm that removes Mixup and consistency parts as well.
From Table 10, we can observe that both SoLar and PRODEN benefit from Mixup and consistency regularization techniques.
In contrast to the relatively complicated contrastive learning modules in PiCO, which may be not directly suitable for the long-tailed setup, our simpler design can also alleviate the representation dilemma of PLL.
C.5 Results with Non-Uniform Data Generation
In reality, some labels may be more analogous to the ground-truth than others, and CIFAR100-H-LT is exactly one of the cases.
Following [7, 22], we further test SoLar with a non-uniform data generation process, with the following flipping matrix:
[Equation]
where each entry denotes the probability of a label being a candidate.
From Figure 3 (c), we evaluate SoLar on CIFAR10-LT with an imbalance ratio γ = 100 .
It can be observed that SoLar still outperforms the baselines, which further validates its strong disambiguation ability.
C.6 Results on Real-world Partial-Label Learning Datasets
In this section, we test the performance of SoLar on four classical real-world datasets 3 , including
Lost, Bird Song, Soccer Player and Yahoo!News.
As shown in Table 11, these datasets are naturally
[Table] [Table]
imbalanced, which highlights the motivation of our work.
In particular, the Soccer Player dataset has an extremely severe imbalanced ratio of 954.33.
Thus, the evaluation protocol of previous works, i.e. , uniformly splitting a testing set, is unrealistic on these real-world datasets.
To this end, we propose a (roughly) balanced testing set sampling rule as follows: i) if one label is associated with ≥ 200 data points, we uniformly select 100 samples for testing; ii) otherwise, we uniformly sample half of the data for testing.
For performance comparisons, we additionally compare SoLar with two more PLL methods that are tailored for tabular data: IPAL [61] is a graph-based PLL method that propagates candidate labels to recover label confidences; PLDA [62] is a feature-selection-based method that maximizes the mutual-information-based dependency between features and labels and we choose
PL-SVM [32] as the base learner.
We disable mixup training and consistency regularization as they are not applicable to tabular data.
As shown in Table 12, SoLar achieves comparable or better performance to all the baselines under the conventional uniform testing set splitting setup.
When the testing set is roughly balanced, SoLar obtains the best performance on all four datasets.
These results further highlight the superiority of SoLar on the imbalanced PLL problem.
D Societal Impact
In this section, we briefly discuss several societal impacts of our work.
The most obvious merit of our study is to reduce the cost of annotation by enabling coarse-grained labeling.
This is a two-edged sword for the community.
On the one hand, non-expert annotators can be employed for crowdsourcing labeling.
On the other hand, if the partial-label learning paradigm is widely applied, the need for precisely annotated data would be significantly reduced, which may cause potential employment destruction as a consequence of reducing the need for human annotators.
Another potential application of our method is data privacy.
For instance, we may ask respondents to answer some private information when collecting some survey data.
The candidate set-style labeling enables the respondents to exclude several wrong answers, which would be more privacy-friendly.