Towards Practical Few-Shot Query Sets: Transductive Minimum Description Length Inference
Abstract
Standard few-shot benchmarks are often built upon simplifying assumptions on the query sets, which may not always hold in practice.
In particular, for each task at testing time, the classes effectively present in the unlabeled query set are known a priori, and correspond exactly to the set of classes represented in the labeled support set.
We relax these assumptions and extend current benchmarks, so that the query-set classes of a given task are unknown, but just belong to a much larger set of possible classes.
Our setting could be viewed as an instance of the challenging yet practical problem of extremely imbalanced K -way classification, K being much larger than the values typically used in standard benchmarks, and with potentially irrelevant supervision from the support set.
Expectedly, our setting incurs drops in the performances of state-of-the-art methods.
Motivated by these observations, we introduce a P rim A l D ual Minimum D escription LE ngth ( PADDLE ) formulation, which balances data-fitting accuracy and model complexity for a given few-shot task, under supervision constraints from the support set.
Our constrained MDL-like objective promotes competition among a large set of possible classes, preserving only effective classes that befit better the data of a few-shot task.
It is hyperparameter free, and could be applied on top of any base-class training.
Furthermore, we derive a fast block coordinate descent algorithm for optimizing our objective, with convergence guarantee, and a linear computational complexity at each iteration.
Comprehensive experiments over the standard few-shot datasets and the more realistic and challenging i-Nat dataset show highly competitive performances of our method, more so when the numbers of possible classes in the tasks increase. Our code is publicly available at https://github.com/ SegoleneMartin/PADDLE .
Introduction
The performance of deep learning models is often seriously affected when tackling new tasks with limited supervision, i.e., classes that were unobserved during training and for which we have only a handful of labeled examples.
Few-shot learning [1, 2, 3] focuses on this generalization challenge, which occurs in a breadth of applications.
In standard few-shot settings, a deep network is first trained on a large-scale dataset including labeled instances sampled from an initial set of classes, commonly referred to as the base classes.
Subsequently, for novel classes, unobserved during the base training, supervision is restricted to a limited number of labeled instances per class.
During the test phase, few-shot methods are evaluated over individual tasks, each including a small batch of unlabeled test samples ( the query set ) and a few labeled instances per novel class ( the support set ).
The transduction approach has becomes increasingly popular in few-shot learning, and a large body of recent methods focused on this setting, including, for instance, those based on graph regularization [4, 5], optimal transport [6, 7], feature transformations [8, 9], information maximization [10, 11, 12] and transductive batch normalization[13, 2], among other works [14, 15, 9, 16, 17].
Transductive few-shot classifiers make joint predictions for the batch of query samples of each few-shot task, independently of the other tasks.
Unlike inductive inference, in which prediction is made for one testing sample at a time, transduction accounts for the statistics of the query set of a task 4 , typically yielding substantial improvements in performance.
On standard benchmarks, the gap in classification accuracy between transductive and inductive few-shot methods may reach 10% ; see [11], for example.
This connects with a well-known fact in classical literature on transductive inference [19, 20, 21], which prescribes transduction as an effective way to mitigate the difficulty inherent to learning from limited labels.
It is worth mentioning that transductive methods inherently depend on the statistical properties of the query sets.
For instance, the recent studies in [22, 10] showed that variations in the class balance within the query set may affect the performances of transductive methods.
Transductive few-shot classification occurs in a variety of practical scenarios, in which we naturally have access to a batch of unlabeled samples at test time.
In commonly used few-shot benchmarks, the query set of each task is small (less than 100 examples), and is sampled from a limited number of classes (typically 5).
Those choices are relevant in practice: During test time, one typically has access to small unlabeled batches of potentially correlated (non-i.i.d.) samples, e.g., smart-device photos taken at a given time, video-stream sequences, or in pixel prediction tasks such as semantic segmentation [12], where only a handful of classes appear in the testing batch.
However, the standard few-shot benchmarks are built upon further assumptions that may not always hold in practice: the few classes effectively present in the unlabeled query set are assumed both to be known beforehand and to match exactly the set of classes of the labeled support set .
We relax these assumptions, and extend current benchmarks so that the query-set classes are unknown and do not match exactly the support-set classes, but just belong to a much larger set of possible classes.
Specifically, we allow the total number of classes represented in the support set to be higher than typical values, while keeping the number of classes that are effectively present in the query set to be much smaller.
Our challenging yet practical setting raises several difficulties for state-of-the-art transductive fewshot classifiers: (i) it is an instance of the problem of highly imbalanced classification; (ii) the labeled support set includes “distraction” classes that may not actually be present in the test query samples; (iii) we consider K -way classification tasks with K much larger than the typical values used in the current benchmarks.
We evaluated 7 of the best-performing state-of-the-art transductive few-shot methods and, expectedly, observed drops in their performance in this challenging setting.
Motivated by these experimental observations, we introduce a Minimum Description Length (MDL) inference, which balances data-fitting accuracy and model complexity for a given few-shot task, subject to supervision constraints from the support set.
The model-complexity term can be viewed as a continuous relaxation of a discrete label cost, which penalizes the number of non-empty clusters in the solution, fitting the data of a given task with as few unique labels as necessary.
It encourages competition among a large set of possible classes, preserving only those that fit better the task.
Our formulation is hyper-parameter free, and could be applied on top of any base-class training.
Furthermore, we derive a fast primal-dual block coordinate descent algorithm for optimizing our objective, with convergence guarantee, and a linear computational complexity at each iteration thanks to closed-form updates of the variables.
We report comprehensive experiments and ablation studies on mini -Imagenet, tiered -Imagenet, and the more realistic and challenging iNat dataset [23] for fine-grained classification, with 908 classes and 227 ways at test-time.
Our method yields competitive performances in comparison to the state-of-the-art, with gaps increasing substantially with the numbers of possible classes in the tasks.
Few-shot setting and task generation
Base training Let D base = { x n , y n } |D base | n =1 denotes the base dataset, where each x n ∈ X base is a sample from some input space X , y n ∈ { 0 , 1 } |Y base | the associated ground-truth label, and Y base the set of base classes.
Base training learns a feature extractor f ϕ :
X → Z , with parameters ϕ and Z a lower-dimensional space.
For this stage, an abundant few-shot literature adopts episodic training, which views D base as a series of tasks (or episodes) so as to simulate testing time.
Then, a meta-learner is devised to produce the predictions.
However, it has been widely established over recent years that a basic training, followed by transfer-learning strategies, outperforms most meta-learning methods [24, 25, 26, 4, 11].
Hence, we adopt a standard cross-entropy training in this work.
Evaluation Evaluation is carried out over few-shot tasks, each containing samples from D test = { x n , y n } |D test | n =1 , where y n ∈ { 0 , 1 } |Y test | , with constraint Y base ∩ Y test = ∅ , i.e., the test and base classes are distinct.
Each task includes a labelled support set S
= { x n , y n } n ∈ I S and an unlabelled query set Q = { x n } n ∈ I Q , both containing examples from classes in Y test .
Using the feature extractor f ϕ trained on base data, the goal is to predict the classes of the unlabeled samples in Q for each few-shot task, independently of the other tasks.
Task generation For a given task, let K denote the total number of possible classes that are represented in the labeled support set S , and K eff the number of classes that appear effectively in unlabeled query set Q (i.e. classes represented by at least one sample).
The standard task generation protocol assumes that the set of effective classes in Q matches exactly the set of classes in S , i.e.
K = K eff ≪ |Y test | , which amounts to knowing exactly the few classes that appear in the test samples.
Then, a fixed number of instances per class are sampled for each query set, forcing class balance.
We relax these assumptions, and propose a setting where the set of effective classes in Q is not known exactly and belongs to a much larger set of possible classes.
First, we allow the total number of possible classes K to be higher than typical values, with K = |Y test | , while keeping the number of effective classes in the query set to be typically much smaller: K eff ≪ |Y test | =
K .
For instance, in our experiments, K can go up to 227 .
This, as expected, results in K -way problems that are more challenging than the standard 5-way tasks used in the few-shot literature.
Secondly, once the effective classes are fixed, and given a budget of images, we sample the query set under the data joint distribution (i.e. uniform draws among all available examples), so that the statistics of the sampled query sets more faithfully reflect the natural distribution of classes.
Figure 1 illustrates our framework (base training, inference and task generation).
Proposed few-shot inference formulation
For a given few-shot task, let Q = { x n } n ∈ I Q and S = { x n , y n } n ∈
I S be two subsets of D test such that Q ∩ S = ∅ .
Let N = | Q | + | S | .
Up to a reordering, we can suppose that I Q = { 1 , . . .
, | Q |} and I S
= {| Q | + 1 , . . .
, N } .
We denote z n = f ϕ ( x n ) the feature vector corresponding to the data sample x n , and u n = ( u n,k ) 1 ≤ k ≤ K ∈ { 0 , 1 } K the variable assigning the data to one of the possible classes in { 1 , . . .
, K } , i.e. u n,k =
if x n belongs to class k and 0 otherwise.
We define the variables giving the class proportions ˆ u =
[Equation] ˆ u k [Equation] 1 ≤ k ≤ K ∈ ∆ K as
[Equation]
where ∆ K is the unit simplex of R K .
We propose to cast transductive few-shot inference as the minimization of an objective balancing data-fitting accuracy and partition complexity, subject to supervision constraints (known labels) from the support set S .
Our approach estimates jointly U = ( u n ) 1 ≤ n ≤ N , which defines a partition of S ∪ Q , and the class prototypes W = ( w k ) 1 ≤ k ≤ K ∈ ( R d ) K through the problem:
[Equation]
Note that, in the second line of (2), we have relaxed the integer constraints on query assignments to facilitate optimization.
Effect of each term The purpose of objective (2) is to classify the data of a few-shot task with as few unique labels as necessary.
The data-fitting term has the same form as the standard K -means objective for clustering, but is constrained with supervision from the support-set labels.
This term evaluates, within each class k , the deviation of features from class prototype w k , thereby encouraging consistency of samples of the same class.
The partition-complexity term implicitly penalizes the number of effective (non-empty) classes that appear in the solution ( K eff ≤ K ), encouraging low cardinality partitions of query set Q .
This term is the Shannon entropy of class proportions within Q : it reaches its minimum when all the samples of Q belong to a single class, i.e., ∃ j ∈ { 1 , . .
, K }
such that ˆ u j = 1 and all other proportions vanish.
It achieves its maximum for perfectly balanced partitions of Q , satisfying ˆ u k = 1 /K for all k .
In practice, this terms promotes solutions that contain only a handful of effective classes among a larger set of K possible classes.
Connection to Minimum Description Length (MDL)
The objective in (2) could be viewed as a partially-supervised instantiation of the general MDL principle.
Originated in information theory,
MDL is widely used in statistical model selection [27].
Assume that we want to describe some input data Z = ( z n ) 1 ≤ n ≤| Q | with a statistical model M , among a family of possible models.
MDL prescribes that the best model corresponds to the shortest description of the data, according to some coding scheme underlying the model.
It balances data-fitting accuracy and model complexity by minimizing L ( Z | M ) +
λ L ( M ) w.r.t M .
L ( Z | M ) measures the code length of the prediction of Z made by M , while
L ( M ) encourages simpler models (Occam’s razor [27]), e.g., models with less parameters.
For instance, in unsupervised clustering, it is common to minimize a discrete label cost as a measure of model complexity [28, 29, 30].
Such a label cost is the number of effective (non-empty) clusters in the solution, and is used in conjunction with a log-likelihood term for data-fitting:
[Equation]
In this expression, M = { ( w k ) 1 ≤ k ≤ K , U } with w k being the parameters of some probability distribution Pr ( z n | k ; w k ) describing the samples in cluster k , while U contains assignments as denoted above.
The discrete model complexity in (3) is commonly used in MDL-based clustering [28, 29, 30], despite the ensuing optimization difficulty.
As this measure is a discrete count of non-empty clusters, it does not accommodate fast optimization techniques.
Typically, it is either handled via cluster merging heuristics (starting from an initial set of clusters) [30] or via combinatorial move-making algorithms [28], both of which are computationally intensive, more so when dealing with large sets of classes.
Our partition-complexity term in (2) can be viewed as a continuous relaxation of the discrete label count in (3) (in the supplemental material, we provide a graphical illustration in the case when K = 2 ).
While penalizing similarly the number of effective classes, it has several advantages over the label count in (3), despite the surprising fact that it is not common in the MDL-based clustering literature, to our best knowledge.
First, it is a continuous function of assignment variables U .
This enables us to derive a fast primal-dual block coordinate descent algorithm (Section 4), with convergence guarantee, and a linear computational complexity at each iteration thanks to closed-form updates of the variables.
Second, it has a clear MDL interpretation: it measures the number of bits required to encode the set of classes, given class probabilities Pr ( k )
=
ˆ u k , ∀ k ∈ { 1 , . . .
, K } .
Following the Kraft-McMillan theorem [27], any probability distribution Pr ( z n | k ; w k ) corresponds to some coding scheme for storing the features of cluster k , and − ln Pr ( z n | k ; w k ) is the number of bits required to represent any feature using coding scheme Pr ( z n | k ; w k ) .
Clearly, our data-fitting term in (2) also fits into this MDL interpretation by assuming that each probability Pr ( z n | k ; w k ) is a
Gaussian distribution with mean w k and covariance fixed to the identity matrix:
[Equation]
Differences with the existing objectives for transductive few-shot inference Unlike most of the existing transductive few-shot methods, our MDL formulation does not encode strong assumptions on the label statistics of the query set (e.g. class balance or/and a perfect match between the query and support classes).
Information maximization methods, such as [10, 11], maximise the confidence of the sample-wise predictions while encouraging class balance.
Optimal transport methods [6, 7] estimate an optimal mapping matrix, which could be viewed as a joint probability distribution over the features and the labels, while imposing a hard class-balance constraint via the Sinkhorn-Knopp algorithm.
Both information-maximization and optimal-transport methods have inherent class-balance bias and, as shown in the experiments, undergo a drastic drop in accuracy when the number of possible classes increases (as this corresponds to highly imbalanced problems).
Prototype rectification [8] transforms the query features so as to minimize the difference between the overall statistics of the query and support sets.
This relies on the assumption that the support and query classes match perfectly.
Therefore, it is not adapted to our setting.
Inspired from graphical models, the Laplacian regularization in [4] is a pairwise label correction (rather than a learning) method, which encourages assigning the same label to query samples that are close in the input space; it does not learn a class representation from the query set (the prototypes being fixed as those of the support set).
Identifying λ through an unbiased probabilistic K-means interpretation of (2) Another interesting view of our few-shot inference objective in (2) can be drawn from the probabilistic K -means objective 5 , well-known in the clustering literature [31, 32, 33].
In fact, probabilistic K -means corresponds to minimizing L ( Z | M ) in (4) as a generalization of K -means, which corresponds to the particular Gaussian choice in (5).
It is a well-known that probabilistic K -means has a strong bias towards balanced partitions [31, 33].
The objective can be decomposed to reveal a hidden term promoting class balance.
Using Bayes rule Pr ( k | z n ; w k ) ∝ Pr ( z n | k ;
w k ) Pr ( k ) , empirical estimates of marginal class probabilities Pr ( k )
=
ˆ u k , and the choice of Pr ( z n | k ; w k ) in (5), we have:
[Equation]
where c = stands for equality, up to an additive constant independent of optimization variables.
Minimizing the last term in the second line of (6) has an effect opposite to the model-complexity term in our objective in (2): it reaches its minimum for perfectly balanced partitions.
Therefore, our objective in (2) is a way to mitigate such a bias in K -means, allowing imbalanced partitions.
This suggests setting λ = | Q | to compensate for the hidden class-balance term in K -means.
This would make our inference hyper-parameter free.
Thus, for our study, we fixed λ =
| Q | in all benchmarks, without optimizing the parameter λ via validation.
Primal-Dual Block-Coordinate Descent Optimization
We derive a block coordinate descent algorithm for our formulation in (2).
At each iteration of our algorithm, the minimization steps are closed-form, with a linear complexity in K and N .
We emphasize that, in problem (2), the minimization over U could not be carried in closed-form, even for fixed prototypes W , and the simplex constraints are difficult to handle.
One straightforward iterative solution to tackle (2) would be to deploy a projected gradient descent algorithm.
However, as shown in the comparisons in Section 5, this strategy is computationally demanding and does not yield better classification results than the algorithm we propose below.
Primal-Dual formulation We transform problem (2) into an equivalent optimization problem by introducing a dual variable.
Let us first define the linear operator A : ( R K ) N −→ R K , which maps the probabilities to the proportions of the classes, i.e. A : U (cid:55)→ (ˆ u k ) 1 ≤ k ≤ K .
Then, problem (2) can be re-written as follows:
[Equation]
where H is the negative entropy function on the positive orthant of R K : H ( x )
= (cid:80)
Kk =1 φ ( x k ) for x = ( x k ) 1 ≤ k ≤ K ∈ R K , with
[Equation]
We can now appeal to the concept of Fenchel-Legendre conjugate function.
For a given convex function f , its conjugate function, denoted by f ∗ , is defined as f ∗ ( V ) =
sup U ( ⟨ V , U ⟩− f ( U )) , where
V is a dual variable [34, Def. 13.1] and ⟨· , ·⟩ denotes the Euclidean scalar product on R K × N .
Since function H is a proper, lower semicontinuous, convex function, according to the Fenchel-Moreau theorem [34, Thm. 13.37], the equality H =
( H ∗ ) ∗ holds.
Hence, it follows that
[Equation]
Plugging (8) into (7), and using the expression of H ∗ [35, Ex. 3.21], one obtains the following minimization problem with respect to the primal variable U , the weights W and the dual variable V :
[Equation]
Handling the simplex constraint To deal efficiently with the simplex constraint in (9), we add an entropic barrier on soft assignment variables u n , leading to the following modified problem:
[Equation]
The last term in (10) acts as a barrier for imposing constraints u n ≥ 0 and, at each iteration, yields closed-form updates of both the dual variables for constraints (cid:80)
Kk =
1 u n,k = 1 and assignments U .
This simplifies the iterative constrained-minimization steps over U in our algorithm below through simple closed-form softmax operations (derivation details provided in supplemental material).
Block coordinate descent algorithm To minimize the cost function in (10), we pursue a block coordinate descent approach [36, 37], which is guaranteed to converge; see Proposition 1 below.
At each iteration, we successively minimize the objective with respect to the variables U , V , and W respectively.
To do so, we use the adjoint operator of A , which we denote A ∗ .
Our algorithm is detailed in Algorithm 1.
Initialize W (0) as the prototypes computed on the support, and V (0)
= 0 .
for ℓ = 1 , 2 , . . .
, do
[Equation]
Convergence guarantees In addition to its practical advantages in terms of implementation, our algorithm benefits from the convergence guarantee described in Proposition 1.
Proposition 1.
The sequence (cid:0) U ( ℓ ) , W ( ℓ ) , V ( ℓ ) (cid:1) ℓ ∈ N ∗ generated by Algorithm 1 is bounded.
Moreover, any of its cluster points is a critical point to the minimization problem in (10) .
A detailed proof of Proposition 1 is provided in the supplementary material.
Experiments 5.1 Experimental details
Datasets.
We deployed three datasets for few-shot classification: mini -Imagenet [38], tieredImagenet [18], and i-Nat [23].
A subset of the ILSVRC-12 data [38], mini -Imagenet is a standard few-shot benchmark, with 60 , 000 color images of size 84 × 84 pixels [3].
It contains 100 classes, each represented with 600 images.
We followed the standard split of 64 classes for base training, 16 for validation, and 20 for testing [39, 25].
The tiered -Imagenet is another standard few-shot benchmark, which is a larger subset of ILSVRC-12, with 608 classes and a total of 779 , 165 color images of size 84 × 84 pixels.
We used a standard split of 351 classes for base training, 97 for validation, and 160 for testing.
Finally, the more realistic and challenging dataset i-Nat hasclasses.
We follow the split from [25, 4], with 227 ways at test-time.
Task generation We build the few-shot tasks as follows.
Let s denotes the number of shots. We constitute the support set by sampling s images according to the uniform distribution for each of the K possible classes of the test set (i.e. K = |Y test | = 20 , 160 , and 227 for mini -ImageNet, tiered -ImageNet and i-Nat , respectively).
For the query set, we first randomly pick K eff < K classes among the K possible classes. We then randomly choose | Q | samples among the images that belong to the K eff classes but do not appear in the support set. The right-hand side of Figure 1 depicts an example of a task, with the class color coding indicating whether a support class appears in the query set or not. The results presented below are obtained by fixing | Q | = 75 , as done in the literature.
In the tables, we use a fixed K eff =5, but we also present the results over a larger range in Fig.
Given the difficulty of K -way tasks, with K large, we consider 5-, 10- and 20-shot supervision to form the support set.
Following standard evaluation, we report the accuracy, averaged across 10 , 000 tasks.
Hyper-parameters We emphasize that PADDLE inference does not require any hyperparameter tuning since λ in objective (10) is set to the number of samples in the query set Q .
This is typically not the case for other few-shot methods.
For the tuning phase of those methods, we have followed the protocol of [10], which for each dataset, uses a 5-way, 5-shot scenario on the corresponding validation set.
For i-Nat , because no validation split is provided, we reuse the hyper-parameters obtained from tiered -ImageNet.
Feature extraction To ensure the fairest comparison of methods, we use the pretrained checkpoints provided by the authors of [11].
The models are trained on D base via a standard cross-entropy minimization with label smoothing.
The label smoothing parameter is set to 0 . 1 , for 90 epochs, using a learning rate initialized to 0 . 1 and divided by 10 at epochs 45 and 66 .
We use batch sizes of 256 for
ResNet-18 and of 128 for WRN28-10.
The images are resized to 84 × 84 pixels, both at training and evaluation time.
Color jittering, random croping, and random horizontal flipping augmentations are applied during training.
[Table] BD-CSPN [8] 49. PT-MAP [7] 25. LaplacianShot [4] 60.
TIM [11] 66.
α -TIM [10] 63.
PADDLE (ours) 63.
Baseline [24]
WRN2858.
LR+ICI [15] 57.
BD-CSPN [8] 51.
PT-MAP [7] 26.
LaplacianShot [4] 64.
TIM [11] 71.
α -TIM [10] 68.
PADDLE (ours) 62.
Results are averaged across 10,000 tasks.
Results marked with ’-’ were intractable to obtain.
5.2 Results
Main results We compare the performances of our method with state-of-the-art few-shot methods.
Our first experimental setting consists in fixing the number of effective classes K eff to 5.
In Table 1, we evaluate the accuracy on mini and tiered -ImageNet for 5, 10, and 20 shots, while results on i-Nat are displayed in Table 2.
Note that the i-Nat dataset comes with a unique support set for all tasks, with a varying number of shots (labeled samples) per class.
Therefore, for i-Nat , we present the results separately.
In the second experiment displayed in Figure 2 and 3, we plot the accuracy as a function of K eff on mini , tiered -ImageNet, and i-Nat for 5, 10, and 20 shots.
Both experiments show that PADDLE is very competitive with the state-of-the-art methods on the proposed new practical few-shot setting.
More importantly, the gap between PADDLE and the second best algorithm increases significantly with (i) the number of shots and (ii) the number of possible classes K , when going from the dataset mini , which has a small number of test classes ( K = 20 ), to tiered ( K = 160 ) and finally to i-Nat ( K = 227 ).
From the first point, one may conclude that the proposed MDL formulation benefits better from additional support supervision.
We hypothesize to this is due to its versatility, as it does not encode strong assumptions on the label statistics of the query set.
Also, the second point could be explained along the same line.
For instance, methods relying on a class-balance assumptions, such as TIM-GD or PT-MAP, observe a drastic drop in accuracy as K increases, even falling below the inductive Baseline.
This is expected because large values of K corresponds to highly imbalanced classification problems for the query sets.
For example, K eff =5 effective classes on tiered corresponds to a relatively more drastic form of class imbalance than on mini , as it would mean only 5 / 160 classes are represented in Q , versus 5 / 20 .
In contrast, PADDLE can deal with strongly imbalanced situations.
Note that BD-CSPN [8] is the only baseline whose performance increases with K eff . This method is also among the most affected when the class overlap between the query and support sets decreases, i.e., when K eff gets smaller (left side of the plots in Figure 2) and/or K gets bigger (e.g. tiered-ImageNet, with K = 160). This behaviour might be due to the fact that BD-CSPN encodes a strong prior, assuming the support and query classes match perfectly.
[Figure]
Method i-Nat ( K =227) Baseline [24] 58. BD-CSPN [8] 57.
PT-MAP [7] 6. LaplacianShot [4] 43.
TIM [11] 37.
α -TIM [10]
[Table]
PADDLE (
[Table]
[Figure]
[Figure]
Ablation on the objective We hereby ablate on the importance of the partition-complexity term in Eq.
Removing this high-order term yields a partially-supervised version of K -means, which can be optimized effortlessly through iterative closed-form alternating steps.
We provide the comparison between this approach ( without the partition-complexity term) and PADDLE ( with the partition complexity term) in Table 3.
On can observe that PADDLE systematically outperforms its K -Means counterpart, with absolute differences in accuracy reaching up to 30% + on challenging scenarios. Partition complexity term in Problem [Table]
Without
[Table]
With [Table] Ablation on the optimization procedure As a second ablation, we propose to compare our proposed Alternating Minimization Algorithm ?? to a straightforward first-order approach.
More precisely, as a comparison, we directly optimize Eq.
(2) through Projected Gradient Descent, denoted as PGD , where a simplex projection step [40] takes place after every iteration to ensure that the simplex constraints on U remains satisfied.
For PGD , we use Adam [41] with α = 0 .
We found this to be the highest learning rate that leads to convergence.
We also add the convergence time of the second best competing method, α -TIM.
Results are provided in Fig. 4.
PGD is much slower than PADDLE , with a ratio between run times neighboring an order of magnitude.
Additionally, as hinted from the last points of the PGD curve, oscillations seem to occur, indicating that a more sophisticated learning rate policy would be necessary to achieve a clear convergence.
[Figure]
( ‘ + 1 ) −
[Figure] [Figure]
Conclusion and limitations
We presented a practical few-shot setting where the number of candidate classes can be much larger than the number of classes that appear effectively in the query set.
Our setting is an instance of highly imbalanced classification, with large numbers of ways and potentially irrelevant supervision from the support set.
We observed much higher gaps across transductive few-shot methods, some of which fall below the simple inductive baseline.
As a solution, we introduced PADDLE , which casts this challenge as a partially-supervised MDL partitioning problem, interpreting the number of unique classes found as a measure of model complexity.
PADDLE is hyperparameter-free, and remains competitive over various state-of-the-art methods, settings and datasets, without the need for any tuning.
However, we do not advocate our method as the one-fits-all, ultimate solution to the extremely challenging few-shot problem.
For instance, PADDLE would not be the best performing method in situations where K eff would approach K .
In applications where one has knowledge about the label statistics of the query set (e.g. class balance), other methods encoding such a knowledge could be more appropriate.
More generally, our setting can be seen as a particular case of class-distribution shift between the support and query sets.
Interesting future works could complement our current study by overlaying other forms of shifts, e.g. feature shifts [42, 43], to our setting.