O N THE I MPORTANCE OF F IRTH B IAS R EDUCTION IN F EW -S HOT C LASSIFICATION
A BSTRACT
Learning accurate classifiers for novel categories from very few examples, known as few-shot image classification, is a challenging task in statistical machine learning and computer vision.
The performance in few-shot classification suffers from the bias in the estimation of classifier parameters; however, an effective underlying bias reduction technique that could alleviate this issue in training few-shot classifiers has been overlooked.
In this work, we demonstrate the effectiveness of
Firth bias reduction in few-shot classification.
Theoretically, Firth bias reduction removes the O ( N − 1 ) first order term from the small-sample bias of the Maximum Likelihood Estimator.
Here we show that the general Firth bias reduction technique simplifies to encouraging uniform class assignment probabilities for multinomial logistic classification, and almost has the same effect in cosine classifiers.
We derive an easy-to-implement optimization objective for Firth penalized multinomial logistic and cosine classifiers, which is equivalent to penalizing the cross-entropy loss with a KL-divergence between the uniform label distribution and the predictions.
Then, we empirically evaluate that it is consistently effective across the board for few-shot image classification, regardless of (1) the feature representations from different backbones, (2) the number of samples per class, and (3) the number of classes.
Finally, we show the robustness of Firth bias reduction, in the case of imbalanced data distribution.
Our implementation is available at https://github.com/ehsansaleh/firth_bias_reduction.
I NTRODUCTION
Few-shot image classification is the practice of learning accurate classifiers using a small number of labeled samples (Fei-Fei et al., 2006; Vinyals et al., 2016; Wang and Hebert, 2016;
Finn et al., 2017; Snell et al., 2017; Wang et al., 2020).
It has a wide range of applications from face and gesture recognition (Pfister et al., 2014) to visual navigation in robotics (Finn et al., 2017).
Essentially, modern few-shot classification methods can be viewed as a combination of learning (1) a strong feature representation through a backbone network (e.g., Verma et al.
; Gidaris et al.
Tian et al.
(2020)), and (2) an accurate small-sample classifier (e.g.
, Wang and Hebert (2016); Chen et al.
Therefore, two key questions arise in few-shot image classification: (1) How can we obtain a strong feature representation?
and (2) How can we train accurate classifiers using a small number of samples?
There have been many existing methods addressing the former question using a host of different techniques (Snell et al., 2017; Rusu et al., 2019; Verma et al., 2019; Mangla et al., 2020).
Here we focus on the less-explored second question.
For our purposes, we will use standard feature representations and methods for training the backbone model for few-shot classification.
That still leaves us with a severe classifier problem than most people realize.
There is a substantial difficulty with training a classifier using a small number of samples.
In particular, with very few samples, standard classification machinery is biased.
In other words, although the Maximum Likelihood Estimators (MLEs) are statistically consistent and asymptotically normal (Fahrmeir and Kaufmann, 1985), it is well-established that MLEs are biased for a small number of N samples, with bias of O ( N − 1 )
(Cox and Snell, 1968; Box, 1971; Whitehead, 1986; Firth, 1993).
Since common logistic regression models are a type of MLEs, they are also biased (Schaefer,
; Cordeiro and McCullagh, 1991; Firth, 1993; Steyerberg et al., 1999).
Such biases increase the error rate of the few-shot trained classifiers, and so are important in few-shot learning.
In fact, there is a standard solution for bias prevention by modifying the ordinary MLEs – known as Firth’s Penalized Maximum Likelihood Estimator (PMLE) (Firth, 1993).
In the case of the exponential family of distributions, Firth has a simplified form that penalizes the likelihood by Jeffrey’s invariant prior (Firth, 1993; Poirier, 1994), which is proportional to the determinant of the Fisher Information Matrix F .
For logistic and cosine classifiers (Chen et al., 2019) which are widely used in few-shot classification, since they belong to the exponential family, Firth bias reduction can be further cast as adding a log-determinant penalty ( log(det( F )) ) to the cross-entropy loss.
While such standard strategies can control the bias in classifiers trained with very few samples, they have not been utilized in few-shot image classification tasks.
In this paper, we show that using Firth bias reduction produces reliable improvements in a wide range of circumstances .
We achieve this by deriving a simplified yet effective Firth formulation that penalizes the Kullback–Leibler (KL) divergence between the uniform distribution of classes and the predictions, for both multinomial logistic regression models and cosine classifiers.
Note that common regularization techniques (such as L2-regularization and label smoothing (Szegedy et al., 2016)) cannot reduce the estimation bias of classifier weights in small-sample regimes (Liu et al., 2020) as the Firth penalty does; these regularization techniques are mainly used to control model complexity of deep neural networks for training feature extractor backbones in large-sample regimes .
More concretely, our results indicate that the improvements produced by Firth bias reduction for few-shot image classification tasks are consistent across the board (1) on a wide range of feature representations, (2) with both balanced and imbalanced data, (3) for both single-layer and multi-layer classifiers, (4) for both logistic and cosine classifiers, and (5) over multiple datasets and problems.
Importantly, we found Firth bias reduction to consistently yield statistically significant and positive improvements , and we did not observe any performance penalty for utilizing it.
Such improvements are on the order of 0.5-2.5% and up to 3% in challenging tasks with large number of classes.
Our main contributions include (1) deriving a generalized expression for Firth bias reduction in fewshot multinomial logistic regression models, and providing geometrical insight into its effect on the classification probability space; (2) evaluating the efficacy of the Firth penalized multinomial logistic model in few-shot scenarios, both with balanced and imbalanced data distributions; (3) showing that
Firth bias reduction can be extended beyond typical logistic models, and can be successfully adopted in cosine classifiers; and (4) providing an empirical comparison of Firth bias reduction with common regularizers such as L2 and label smoothing.
B ACKGROUND
Mathematical Notations:
In this work, we assume a multinomial logistic regression model for the classifier, with a total of J + 1 classes { 0 , 1 , 2 , · · · , J } .
The logistic regression weights for class j is denoted as β j ( 1 ≤ j ≤ J ).
The class j = 0 is the reference class with zero logistic regression weights.
We assume to have a total number of N samples D
= { ( x 1 , y 1 ) , · · · , ( x N , y N ) } .
The i th target y
i is the one-hot encoding of the i th label.
The assignment probability of the i th sample to class j is denoted as P i,j :
[Equation]
The likelihood of the sample set D given the weights β is denoted as Pr( y | x ; β ) :
[Equation]
where 1 [ · ] denotes the binary indicator function.
Therefore, the logistic log-likelihood function L logistic is defined as
[Equation]
[Figure]
[Figure]
The Fisher Information Matrix (FIM) is defined as the Hessian of the negative log-likelihood function
L logistic :
[Equation]
The Maximum Likelihood Estimator (MLE) for logistic regression is defined as
[Equation]
The dimension of the feature space is denoted as d in the derivations.
U [ a,b ] denotes the (discrete) uniform distribution in the [ a, b ] interval.
The cross-entropy and the KL-divergence of distributions p, q are defined as
[Equation]
Table A2 in the Appendix summarizes these notations.
Small-Sample Bias of MLE:
Assume β ∗ is the true generative parameter.
When the sample size N is small, it is shown that the MLE bias b (ˆ β MLE ) :=
E [ˆ β MLE − β ∗ ] is non-zero and of O ( N − 1 )
(Cox and
Snell, 1968).
Therefore, while MLE is unbiased as N → ∞ , it is inaccurate for few-shot learning.
Firth Bias Reduction for MLE: Firth’s PMLE (Firth, 1993) is a modification to the ordinary MLE, which removes the O ( N − 1 ) term from the small-sample bias.
In particular, Firth has a simplified form for the exponential family .
When Pr( y | x ; β ) belongs to the exponential family of distributions, the effect is to penalize the likelihood by Jeffrey’s invariant prior (Poirier, 1994), which is proportional to the determinant of F ( det( F ) ).
Logistic and cosine classifiers (Chen et al., 2019) – the widely-used classification models in few-shot learning – belong to the exponential family; so for them, Firth bias reduction simplifies to adding a penalty ( log(det( F )) ) to the cross-entropy loss.
In what follows, we will derive a further simplified yet effective Firth formulation for logistic and cosine classifiers, which is computationally more efficient, deals with the case when det( F ) = 0 , and generalizes to multinomial distributions.
MLE vs. Firth’s PMLE for a Simple Case: To demonstrate the extent of the MLE bias and how Firth’s PMLE removes the leading O ( N − 1 ) bias term, we simulated data from the geometric distribution with probability of success β = 0 .
5 as its only parameter.
The geometric experiment was chosen since a closed-form solution for the MLE and Firth’s PMLE can be derived.
Given the samples ( y 1 , · · · , y N ) from the geometric distribution, the sample mean is ¯ y = 1 N (cid:80)
y
i and we have ˆ β MLE = 1 ¯ y and ˆ β Firth = N − 1 N ¯ y − 1 .
Note that since the sample mean is noisy, the MLE suffers from a noisy denominator, making it biased.
Figure 1 shows that ˆ β
Firth is close to the true parameter β ∗ for all sample sizes, whereas ˆ β MLE has a significant bias away from β ∗ for small N .
To further validate that the MLE bias is indeed of O ( N − 1 ) , we plotted the MLE bias against the sample size in the log-log scale in Figure1, which shows that it is closely following a line with a negative unit slope.
F IRTH B IAS R EDUCTION IN L OGISTIC AND C OSINE C LASSIFIERS
In logistic models, the penalized likelihood function proposed by Firth is equivalent to imposing
Jeffreys’ prior (Poirier, 1994) on the parameters and making a maximum a posteriori estimation.
In particular, Firth bias reduction encourages models with “ large ” F by multiplying the likelihood by det( F ) .
This penalty degenerates when det( F ) = 0 .
We work in the highest dimensional subspace where F has full rank, and use det( F | r ) to denote the product of all r non-zero eigenvalues of F , obtaining
[Equation]
(7) where Z is a normalization constant and det( F | r ) 1 / 2 is the Jeffery’s prior.
Taking the log of both sides yields the log-posterior as a sum of the logistic log-likelihood function and the Firth bias reduction term:
[Equation]
where we have
[Equation]
The definition of L Firth was left ambiguous up to a constant with respect to β to facilitate the Firth bias reduction term’s interpretation and avoid the definition of similar terms.
Furthermore, λ controls for the impact of the Firth term on the outcome relative to L logistic .
We then apply a series of derivation steps to simplify Equation (9), which are left to Section A in the Appendix.
Finally, the Firth bias reduction term can be expressed as
[Equation]
For the cosine classifier, the proof that log(det( F )) simplifies to Equation (10) involves straightforward manipulation of the proof for the logistic classifier, and is left to Section B in the Appendix.
The normalization of the β j weights in the cosine classifier turns into a pure scale term in the optimization, and for a cosine classifier, scaling of the β j weights does not affect predictions.
Therefore, this term should be ignored, and the bias reduction term effectively becomes the same as Equation (10).
Interpreting the Firth Bias Reduction for Logistic Models: It is well known that Jeffery’s prior shrinks the parameter estimates towards zero, which is equivalent to encouraging uniform class assignment probabilities (Firth, 1993;
Bull et al., 2002).
We take an alternative approach to reach the same conclusion in the following.
By re-arranging Equation (10), one can see the (cid:80) Jj =0 log( P i,j ) term as a scaled average over a uniform distribution of classes:
[Equation]
Therefore, we can abuse the notation, and redefine the coefficient λ and the constant to have
[Equation]
This means that by dropping the constants, the optimization objective L can be rewritten as
[Equation]
Since the KL-divergence and the FIM define an information geometry and a Riemanian metric on probabilistic measure spaces (Nielsen, 2020), a geometrical insight into Equation (13) is provided in
Figure A5 in the Appendix as well.
Firth Bias Reduction vs. Common Regularization: While this insight brings Firth bias reduction closer to the common regularization techniques (e.g., L2-regularization), it is worth noting that (1) common regularization techniques mainly focus on controling model complexity, instead of reducing small-sample estimation bias; (2) Firth bias reduction operates on a much lower-dimensional target distribution space, unlike L2 which operates in the high-dimensional parameter space; (3) Firth uses the same kind of metric as the logistic loss; and (4) Firth bias reduction is dimensionally consistent like the Natural Gradients (Amari, 1998; Pascanu and Bengio, 2013), whereas L2 is not.
Firth Bias Reduction vs. Label Smoothing: Notice that the original form of label smoothing used for training in large-sample regimes (Szegedy et al., 2016) has the same formulation as the simplified
Firth penalty term in Equation (12) for multinomial logistic classifiers .
However, Firth bias reduction is inherently different in that it reduces the classifier estimation bias in the small-sample regimes, whereas label smoothing penalizes over-confident predictions when training deep neural networks with large amounts of samples.
Generally, the Firth bias reduction term (i.e., log(det( F )) ) is not the same as the label smoothing penalty (i.e., D KL ( U ∥ P i ) ) for deep neural networks.
Additional analysis and empirical comparisons are provided in Section 4.4 and Section G in the Appendix.
E XPERIMENTAL R ESULTS
Here we show that for a wide range of experiments, Firth bias reduction is a reliable source of small yet useful improvements in the performance.
Since we report improvements for a wide range of methods and settings, the absolute accuracy improvements were reported.
These absolute improvements sometimes constitute significantly to the baseline accuracy in terms of relative importance.
Datasets: We perform experiments on four widely-used and publicly available benchmarks: miniImageNet (Vinyals et al., 2016), CIFAR-FS (Bertinetto et al., 2019), tiered-ImageNet
(Ren et al., 2018), and CUB (Wah et al., 2011).
Each dataset consists of non-overlapping base, validation, and novel classes.
The detailed class splits are described in Section D in the Appendix.
Following the standard practice (Chen et al., 2019), we train feature backbones on base classes, cross-validate bias reduction coefficients on validation classes, and train classifiers and measure test accuracy over multiple trials on novel classes.
Implementation Details: Details regarding the setup, implementation, statistical significance, and reducing the effect of randomized factors are covered in Appendix Section D.
Baselines and Evaluation Metric: Non-penalized classifiers are used as the baseline in all experiments to compare Firth bias reduction and L2-regularization.
Absolute accuracy improvements over the baseline averaged across multiple trials are used as the evaluation metric in all experiments.
Relative improvements are also shown in the Appendix, which demonstrate similar behaviors.
Summary of Results: Section 4.1 shows the efficacy of Firth bias reduction on standard feature backbones (ResNets with varying depth) and single-layer logistic classifiers.
Section 4.2 shows and argues that Firth bias reduction outperforms L2-regularization on few-shot classification tasks.
Next, we investigate the driving factor in Firth’s improvement in Section 4.3, and show evidence for the efficacy of Firth’s bias suppression property.
We also show that Firth bias reduction can be effectively, and without any modifications, applied to imbalanced data distribution settings.
Section 4.4 compares Firth bias reduction against label smoothing variants, and shows that Firth outperforms label smoothing.
In Section 4.5 Firth bias reduction is applied to modern few-shot methods with advanced feature backbones (i.e., WideResNet trained with strong regularization (Mangla et al., 2020)) and cosine classifiers.
Experiments with additional feature backbones (DenseNet and MobileNet (Wang et al., 2019)) are included in Appendix Section F. Finally, Section 4.6 demonstrates that Firth bias reduction produces reliable improvements over the state-of-the-art feature calibration method (Yang et al., 2021).
Our collective results clearly indicate a consistent pattern of improvements over a large array of (1) feature representations, (2) datasets, (3) classification ways, (4) number of shots, (5) types of classifiers, and (6) with both balanced and imbalanced data distributions.
Number of Shots 0.
[Figure]
Firth Bias Reduction
ResNet
ResNet ResNet
ResNet ResNet
Number of Shots 0.
( A
CC ) %
L2 Regularization
ResNet ResNet
ResNet ResNet
ResNet mostly statistically insignificant this experiment,
Absolute and delta accuracies are provided in Table A9.
The same experiments were repeated with a 3-layer classifier instead of a 1-layer classifier in Figure A7 in the Appendix and show results.
4. F IRTH B IAS R EDUCTION I MPROVES THE A CCURACY OF L OGISTIC C LASSIFIERS
Figure 2 summarizes the results.
For all the k -shot tasks and backbones, the average absolute test accuracy improvement is significant and it increases as k increases.
Also, Firth bias reduction does not hurt the 1 and 5 -shot performance, with slight improvements for some backbones.
Extra experiments are included in Appendix Section E.1.
4. L2-R EGULARIZATION IS NOT AS E FFECTIVE AS F IRTH B IAS R EDUCTION
To explore whether the performance improvement pattern for Firth bias reduction can be reproduced by common regularization techniques such as L2, the same experiments were repeated with L2regularized logistic classifiers.
For almost all the k -shot tasks and feature backbones, the average absolute test accuracy improvement is close to zero, as shown in Figure 2.
This suggests that the bias reduction property of Firth plays a significant part in improving the classifier’s performance in the few-shot regime which cannot be achieved by L2-regularization.
4. B IAS OR P RIOR ?
Firth bias reduction clearly helps.
It may be doing so because it is an effective way of suppressing bias.
Alternatively, reinforcing the prior information might be helping.
This section offers evidence that bias suppression is what is important.
We consider data where the class frequencies are imbalanced.
We apply a variant of Firth bias reduction that uses class prior frequencies to this data.
This variant is not as successful as routine Firth bias reduction, suggesting that bias correction is what is important.
One might consider replacing the uniform distribution in the Firth prior with a class probability distribution.
That is, replacing L Firth = − λ · D KL ( U [0 ,J ] ∥ P i ) by L Firth = − λ · D KL ( A ∥ P i ) with A being the imbalanced class distribution, when training a Firth penalized few-shot logistic classifier.
To test this, we designed two schemes to generate imbalanced datasets.
These schemes differ in the imbalanced count vectors used to create the few-shot dataset in validation and novel splits.
The increments in the counts per class were designed to result in two different average counts of 7.
(Scheme 1) and 15 (Scheme 2) over the classes (experimental details are left to Section D in the
Appendix).
For each scheme, the experiments were carried out similar to the “balanced few-shot” case on mini-ImageNet, except that in each trial three classifiers – (1) baseline (not penalized), (2) Firth penalized (KL with the uniform prior), and (3) modified Firth penalized (KL with the imbalanced
R N
R N R N
[Figure]
N
[Figure]
N
[Figure]
5 Average Shots
[Figure]
R N
R N R N
R N R N
[Figure]
CC
[Figure]
% 15 Average Shots
[Figure]
of shots
vary from 2- to 16- in the
The difference of accuracy improvements between the uniform and the non-uniform priors is more striking for the 7.5-average-shot case, which suffers from a greater deal of estimation bias than the 15-average-shot scenario.
This shows that the improvements of Firth bias reduction are driven by its bias suppression property, and are not a mere consequence of imposing a uniform class prior.
The same experiments were repeated with a 3-layer classifier in Figure A10 in the Appendix.
prior) – were trained.
In both schemes, the improvement achieved by the Firth penalized classifier is higher than that of the modified Firth penalized classifier over the baseline for all backbones, as shown in Figure 3.
The difference between the two improvements is more substantial in Scheme 1, where the average of samples per class is less than Scheme 2.
This implies that the Firth penalized classifier is indeed reducing the bias and should not be naively considered as if a prior is simply defined over the class assignment probabilities.
4. F IRTH B IAS R EDUCTION I MPROVES O VER L ABEL S MOOTHING
Even though the original version of label smoothing (Szegedy et al., 2016) has the same formulation as the Firth bias reduction term in Equation (13), there is a family of label smoothing techniques that aim to reduce overfitting and over-confident predictions (Pereyra et al., 2017).
Namely, the confidence penalty (i.e., regularizing D KL ( P i ∥ U ) ) and unigram label smoothing (i.e., regluarizing the
KL-divergence with class priors) are two label smoothing variants, which are shown to outperform the original label smoothing, when training complex networks with large amounts of samples (Pereyra et al., 2017).
Figure 3 shows that Firth bias reduction outperforms unigram label smoothing, and
Table A4 in the Appendix shows that Firth bias reduction yields better results than applying confidence penalty and entropy regularization.
This further suggests that the improvements obtained by Firth are the result of its bias reduction property.
4. F IRTH B IAS R EDUCTION I MPROVES THE P ERFORMANCE OF C OSINE C LASSIFIERS
As cosine classifiers have recently proven to be useful for few-shot classification (Gidaris and
Komodakis, 2018; Chen et al., 2019), we investigated the impact of Firth bias reduction on them.
The difference between a logistic and cosine classifier parameter space is that the parameter space of the cosine classifier is constrained to a unit sphere, therefore it is likely for the Firth bias reduction to improve cosine classifier performance as well.
To test this, we based our experiments on the implementation of the S2M2 R method (Mangla et al., 2020), and used their pre-trained WideResNet28-10 (Zagoruyko and Komodakis, 2016) on the base classes as a strong feature extractor.
We trained
Firth penalized and non-penalized cosine classifiers on the mini-ImageNet, tiered-ImageNet, and
CIFAR-FS datasets.
Few-shot classifiers were trained for varying number of classes, depending on
[Figure]
Number
[Figure] [Figure]
[Figure] [Figure]
Number of Classes 0.
[Figure]
[Figure]
CIFAR-FS 1-shot 5-shot 10-shot
[Figure]
dataset, varying number of per class.
We scheme
Mangla et al
(2020) to generate the k -shot support and query datasets for training and respectively.
As shown in Figure 4, in all datasets, Firth bias reduction improves the performance of 5-, and 10-shot tasks in a monotonically increasing fashion with respect to the number of classes.
This behavior is well represented for tiered-ImageNet, as increasing 5-way to 150-way classification results in more than 2.5% improvement of the average test accuracy.
In the 1 -shot task, the improvement by Firth bias reduction is monotonically decreasing with the number of classes in all the datasets except tieredImageNet, but it does not hurt the performance.
The improvement almost falls within (0.1%-0.5%), (0-0.1%), and (0-0.2%), for mini-ImageNet, CIFAR-FS, and tiered-ImageNet, respectively.
This experiment suggests that not only is Firth bias reduction effective for cosine classifiers, but it also could help with the performance even when strong features are used.
This is well justified as
Firth bias reduction targets the bias introduced in the classifier parameters not the features.
4. C OMPARISON WITH A DDITIONAL S TATE - OF - THE -A RT M ETHOD
We further applied the Firth bias reduction term to the recent state-of-the-art method in few-shot classification (Yang et al., 2021), called Distribution Calibration (DC).
DC computes the closest base classes to each sample in the feature space, and samples artificial examples from a Gaussian distribution centered around the mean of the closest base class features.
To make the setup challenging, we consider data where the base classes are categorically different from the novel classes: i.e., by performing cross-dataset few-shot classification, where the features are trained on the base classes of mini-ImageNet or tiered-ImageNet, and the novel sets are instead sampled from the CUB dataset.
The results are shown in Table 1, and indicate that the Firth bias reduction improves over Yang et al.
We also show the Firth bias reduction improvements on the tiered-ImageNet dataset in
[Table]
Appendix Section H and Table A5.
This further supports the observation that Firth bias reduction yields small but reliable improvements under different methods.
R ELATED W ORK
Bias Reduction of the MLE:
A myriad of statistical work has been proposed to mitigate the smallsample bias of the MLE under different settings (Anderson and Richardson, 1979; Kenward and
Roger, 1997; Bull et al., 2002; Kosmidis and Firth, 2009).
Originally, the asymptotic bias of MLE was shown to be of O ( N − 1 ) , with N being the sample size (Firth, 1993).
To counter such an estimation bias, many approaches have existed.
To name a few, (1) additive penalization terms to the main logistic loss were proposed to reduce the MLE’s bias (Firth, 1993;
Bull et al., 2002; Greenland and
Mansournia, 2015), and (2) some methods have been proposed to directly approximate and remove such a small sample bias (Cox and Hinkley, 1979).
While the latter approach may sound appealing, estimating the MLE’s bias can be impractical.
For instance, in few-shot scenarios, a perfect separation of the classes may be achievable, causing the logistic MLE to be unbounded (Heinze and Schemper, 2002).
On the other hand, the penalization methods do not modify the estimated parameters directly, and instead gently push for a preference towards less biased estimates.
Such penalization methods can be generally applicable to a vast array of models.
Theoretically, Firth’s PMLE reduces the bias by removing the leading O ( N − 1 ) term from the MLE’s bias (Firth, 1993) – a property that does not exist in common regularization techniques such as
L2-regularization.
Furthermore, PMLE of the logistic model has been shown to have smaller variance than MLE as well (Copas, 1988; Kosmidis and Firth, 2009).
Firth’s PMLE has been well studied for binomial logistic regression (Firth, 1993), and applied and tested against other penalization techniques in other fields (Rainey and McCaskey, 2015; Muchlinski et al., 2016; Rahman and Sultana, 2017).
Additional related work on few-shot image classification was left to Appendix Section C.
C ONCLUSION
We show that Firth bias reduction consistently improves the accuracy across the board in few-shot classification regardless of (1) the employed feature backbone, (2) the number of classes and samples, and (3) the dataset and problem setting.
Furthermore, our experiments show that Firth bias reduction can improve the performance of the cosine classifiers, and is applicable to imbalanced few-shot settings without any necessary modifications.
Overall, our evaluations suggest that Firth bias reduction is a useful and general bias reduction tool that has been missing in few-shot classification, and should be incorporated in few-shot classification tasks for accuracy improvements.
A CKNOWLEDGMENT
This work utilized resources supported by (1) the National Science Foundation’s Major Research
Instrumentation program (Kindratenko et al., 2020), grant number and (2) the National
Science Foundation’s Creating Knowledge with All-Novel-Class Computer Vision program, grant number 2106825.
Also, this work made use of the Illinois Campus Cluster, a computing resource that is operated by the Illinois Campus Cluster Program (ICCP) in conjunction with the National Center for Supercomputing Applications (NCSA) and is supported by funds from the University of Illinois at Urbana-Champaign.
A F IRTH B IAS R EDUCTION FOR F EW -S HOT M ULTINOMIAL L OGISTIC R EGRESSION
Table A2 summarizes the notations used throughout the main paper and here.
Given a dataset of N samples D = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , · · · , ( x N , y N ) } , the multinomial logistic model for a total of J +classes can be formulated as
[Equation]
where class j = 0 was chosen as the reference class in the log odds ratio.
In other words, w.l.o.g. we assume β 0 = 0 in this formulation.
Given the decision rule in Equation (A14), we can write
[Equation]
Under this notation, the log-likelihood would be L logistic =
(cid:80) Ni =1 (cid:80)
Jj =0
[ y i =
j ] · log( P i
[Equation]
j ) .
The data matrix X D is given as
[Equation]
Also, X := X D ⊗ I J where the ⊗ operator denotes the Kronecker matrix product, and the Jdimensional identity matrix is denoted as I J .
Firth (1993) has established that the bias of logistic regression can be removed by maximizing the sum of (1) the log-likelihood L logistic and (2) the log-determinant of the Fisher Information Matrix (FIM).
For our purposes, this presents some challenges: we have a few number of samples and the
FIM determinant is zero.
Instead, we use the product of all non-zero eigenvalues of the FIM as its “ amended determinant ”.
To obtain this efficiently, we need to know the specific structure of the FIM.
It is important in what follows that the FIM can be defined as the matrix product
[Equation]
where M is a block-diagonal matrix whose i th diagonal block is denoted as M i .
We leave the definition of X and M i to the “ FIM Formulation for Logistic Regression ” subsection.
Next, we focus on:
• Determinant Amendment and Constant Dropping : Generically, we show that
[Equation]
where det( F | NJ ) is an amended version of det( F ) .
• Efficient Computation of log(det( M i ))
: Next, we show that
[Equation]
Combining these two points will lead us to the simplified Firth bias reduction objective:
[Equation]
Determinant Amendment and Constant Dropping:
Having F = X ⊺ · M · X prompts us to utilize the SVD of X as
[Equation]
Therefore, the FIM can be written as F = V · (cid:0) S ⊺ · K · S (cid:1) · V ⊺ , where K :=
U ⊺ · M · U .
Since V and U are rotation matrices, we can write det( F ) = det( S ⊺ · K · S ) , and det( K ) =
det( M ) =
N (cid:89) i = det( M i ) .
(A22)
[Table]
The Logistic Regression Weights for Class j ( j ∈ { 1 , 2 , · · · , J } )
[Table]
Classification Probability of Sample i
Belonging to Class j
[Table]
The i th Sample’s Soft Classification Probabilities
[Table]
Number of
[Table]
{ ( x
[Table]
y
[Table] [Table]
Logistic Regression Sample Dataset
[Table]
The One-Hot Encoding of the Label y i 1 [ a = b ]
The Binary Indicator Function (i.e., 1 when a = b and 0 otherwise)
F
The Fisher Information Matrix
[Table]
Dimension of the Features
[Table]
The J × J Identity Matrix
A ⊗ B
The Kronecker Product of Matrix A by Matrix B
[Table]
The All Ones Matrix with r Rows and c Columns
[Table]
The Maximum Likelihood Estimator (MLE)
[Table]
The Logistic Log-Likelihood Function
[Table]
The Firth Bias Reduction Function λ
The Firth Bias Reduction Coefficient
CE ( p ∥ q )
The Cross-Entropy of p and q
D KL ( p ∥ q )
The KL Divergence of p and q
U [0 ,J ]
The Uniform Class Assignment Probabilities det( A | r )
The Amended Determinant of the Degenerate Matrix A with at most r
Non-zero Eigenvalues (See Section A in the Appendix)
As we have d > N for most few-shot tasks, the matrix S can be viewed in the following form:
[Equation]
(A23) where ˆ S is a diagonal square matrix.
Since ˆ S ⊺ = ˆ S , we have
[Equation] [Equation]
(A24)
Equation (A24) and det( F ) = det( S ⊺ · K · S ) show why det( F ) is zero.
For mitigation, we replace det( F ) with the product of the non-zero eigenvalues of F , namely det( F | NJ ) , and call it the “ amended determinant ” of F .
Thanks to F = V · (cid:0) S ⊺ · K · S (cid:1) · V ⊺ , even the amended determinants det( F | NJ ) and det( S ⊺ · K · S | NJ ) are the same:
[Equation]
A
[Equation]
[Figure]
Could det( F | NJ ) be zero?
The answer is “ not ” generically; det( M ) is generically positive as we will show det( M i )
= (cid:81)
Jj =
P i,j later in Equation (A28).
Also, det( ˆ S 2 ) > 0 holds with probability 1 for continuous data distributions.
In fact, det( ˆ S 2 ) can only be zero when the data contains linearly dependant samples, which happens with zero probability for non-atomic data distributions.
Therefore, we have
[Equation]
This is the same as Equation (A18): since the log(det( ˆ S 2 )) term is independent of the model’s parameters, we can treat it as an optimization constant and drop it.
Efficient Computation of log(det( M i ))
We define the soft predictions of the i th sample (excluding the reference class) as P i, 1: J :=
[Equation]
[Equation]
⊺ .
Given M
i
’s definition in Equation (A31), we can write
[Equation]
Next, we use the Matrix-Determinant Lemma (Harville, 1998) to compute det( M i ) :
[Equation]
Taking the log will give us Equation (A19)
[Equation] The FIM Formulation for Logistic Regression: Elementary methods established that the FIM of the logistic classifier is composed of J × J block matrices, each with a dimension of d × d (where d is the dimension of the features).
These block matrices can be expressed as
[Equation]
Number of Shots 0.
( A
CC ) % 5-way
ResNet ResNet
ResNet ResNet
ResNet
Number of Shots 0.
( A
CC ) % 10-way
ResNet ResNet
ResNet ResNet
ResNet
This facilitates the expression of the FIM as shown in Equation (A17): We can define X as the
Kronecker product of (1) the data matrix X D and (2) the J -dimensional identity matrix I J :
[Equation]
(A30)
M would then be a block-diagonal matrix whose i th block is defined as ( M i ) j,j = P i,
[Equation]
(1 − P i,j ) ∀ 1 ≤ j ≤ J, ( M i ) j,k = − P
i,j P i,k ∀ 1 ≤ j ̸ = k ≤ J.
(A31)
B F IRTH B IAS R EDUCTION FOR C OSINE C LASSIFIERS
Section A derived the Firth bias reduction term for the logistic classifier as D KL ( U ∥ P i ) .
Here, we generalize this observation to cosine classifiers, and prove that the Firth bias reduction for cosine classifiers reduces down to the same form as the one obtained for logistic classifiers.
By defining the normalization transformation
[Equation]
we have the log-likelihood relation between the cosine classifier and the logistic model:
[Equation]
According to Equation (4) in the main paper and the chain rule, we can write
[Equation]
where A d ( J +1) × d ( J +1) is the Jacobian matrix of T ( β ) with respect to β .
It can be shown that A is a symmetric block-diagonal matrix, whose j th diagonal block is
[Equation]
[Figure]
Firth Bias
[Figure] [Figure]
[Figure] [Figure]
Therefore,
[Equation]
It is obvious that
[Equation]
Therefore, A j,j can be thought as an identity-proportional matrix in the sub-space perpendicular to β j .
Therefore, its amended determinant is
[Equation]
For the plain logistic model, the Firth penalized optimization problem is:
[Equation]
[Figure]
For the cosine classifiers, the Firth penalized optimization problem is:
[Equation]
where we have C := 2 λ ( d − 1) .
By re-parameterizing β = ( u β , s β ) with u β := T ( β ) and s β := (cid:2) ∥ β 0 ∥ , · · · , ∥ β J ∥ (cid:3) , we have
[Equation]
Since u β and s β are independent optimization parameters, this problem can be restated as max u β
[Equation]
Since s β has no effect on the predictions of the model, it can be ignored.
Therefore, we end up with the optimization problem
[Equation]
Essentially, this suggests applying the same bias reduction form to cosine classifiers as the one used for logistic classifiers log(det( F logistic ( u β , T ( x ) , y ))) reduces down to D KL ( U ∥ P i ) as proven in
Section A, and L logistic ( u β ; T ( x ) , y ) is the cross-entropy loss with the true labels.
C A DDITIONAL R ELATED W ORK ON F EW -S HOT I MAGE C LASSIFICATION
As one of the unsolved problems in machine learning, few-shot learning (Miller et al., 2000; Fei-Fei et al., 2006) has attracted growing interest in the deep learning era (Lake et al., 2015; Santoro et al., 2016; Wang and Hebert, 2016; Vinyals et al., 2016;
Snell et al., 2017; Finn et al., 2017; Hariharan and
R N 
 R N
R N 
 R N
R N 1
[Figure]
7.5 Average Shots
[Figure]
R N 
 R N
R N 
 R N
R N 1
[Figure]
15 Average Shots
[Figure]
Girshick, 2017; George et al., 2017; Triantafillou et al., 2017; Edwards and Storkey, 2017; Mishra et al., 2018; Douze et al., 2018; Wang et al., 2018; Chen et al., 2019; Dvornik et al., 2019; Allen et al., 2019; Li et al., 2019a; Yoon et al., 2019; Lifchitz et al., 2019; Li et al., 2019b; Zhang et al., 2020; Ye et al., 2020; Dhillon et al., 2020; Triantafillou et al., 2020; Tian et al., 2020; Dvornik et al., 2020; Yang et al., 2021; Phoo and Hariharan, 2021; Liu et al., 2021; Zhang et al., 2021).
Successful generalization from few training samples requires “inductive biases” or shared knowledge from related tasks (Baxter, 1997), which is commonly acquired through transfer learning, and more recently, meta-learning (Thrun, 1998; Schmidhuber, 1987;
Schmidhuber et al., 1997; Bengio et al., 2013).
By explicitly “learning-to-learn” over a series of few-shot learning tasks (i.e., episodes), which are simulated from base classes, meta-learning exploits accumulated task-agnostic knowledge to few-shot learning problems of novel classes.
Within this paradigm, various types of meta-knowledge has been explored, including (1) a generic feature embedding or metric space, in which images are easy to classify using a distance-based classifier such as cosine similarity or nearest neighbor (Koch et al., 2015;
Vinyals et al., 2016;
Snell et al., 2017; Sung et al., 2018; Ren et al., 2018; Oreshkin et al., 2018); (2) a common initialization of network parameters (Finn et al., 2017; Nichol and Schulman, 2018;
Finn et al., 2018) or learned update rules (Andrychowicz et al., 2016; Ravi and Larochelle, 2017; Munkhdalai and Yu, 2017; Li et al., 2017; Rusu et al., 2019); (3) a transferable strategy to estimate model parameters based on few class examples (Bertinetto et al., 2016; Qiao et al., 2018;
Qi et al., 2018; Gidaris and Komodakis, 2018), or from an initial small dataset model (Wang and
Hebert, 2016; Wang et al., 2017).
Some most recent work (Gidaris and Komodakis, 2018; Chen et al., 2019; Tian et al., 2020) also showed that the performance of these complex models can be matched by simple representation learning on base classes and classifier fine-tuning on novel classes.
D D ATASETS AND A DDITIONAL E XPERIMENT D ETAILS
Datasets: mini-ImageNet consists of 64 , 16 , and 20 classes from ImageNet (Russakovsky et al., 2015) for base, validation, and novel sets, respectively.
Each class contains 600 images of size 84 × 84 .
tiered-ImageNet consists of 351 , 97 , and 160 classes from ImageNet for base, validation, and novel sets, respectively.
Unlike mini-ImageNet, the classes could have varying number of samples in tiered-ImageNet, but the images are of the same size.
CIFAR-FS is a random split of CIFAR-100 (Krizhevsky and Hinton, 2009) with images of size 32 × 32 into 64 , 16 , and 20 classes for base, validation, and novel sets, respectively.
CUB consists of 11,788 images of size 84 ×which are split into 100, 50, and 50 classes for base, validation, and novel sets, respectively.
Setups and Implementation Details: Our experiments fall into two main categories: (1) the standard ResNet (He et al., 2016) feature experiments for logistic classifiers, and (2) more powerful,
[Table]
All Standard Backbone Experiments
[Table]
Balanced Data Experiments
in Sections 4.1 and 4.2 in the
Main Paper
[Table]
Imbalanced Data Experiments in Section 4.3 in the Main
Paper 7.5-Shot Class Distribution
[Table]
Distribution
[Table]
Experiments in
Sections 4.1, 4.2, and 4.3 in the Main Paper with 1-Layer
Logistic Classifier
[Table]
Experiments in
Section E.2 with 3-Layer
Logistic Classifier
[Table]
the number of classifier parameters.
[Table]
state-of-the-art WideResNet (Zagoruyko and Komodakis, 2016) features trained with strong regularization techniques (manifold mixup) and additional self-supervision (Mangla et al., 2020), or further calibrated via feature transformations (Yang et al., 2021), for logistic and cosine classifiers.
[Figure]
For the first category of experiments, we averaged our results over 400 random trials.
An array of 5 different ResNet architectures were trained in this category: ResNet10, 18, 34, 50, and 101, following a simple pipeline in the Pytorch library example 1 .
It is worth noting that we deliberately did not engineer strong features for this category of experiments.
The mini-ImageNet dataset was used for these experiments, and 16-way classification was performed on both the validation and novel classes.
For this purpose, 16 out of 20 novel classes were chosen at random once and fixed for all the evaluations.
We split the validation and novel classes into 90% training and 10% held-out for accuracy evaluation.
The imbalanced data settings were chosen to either have 7.5 or 15 average number of shots per class.
For the second category of experiments, we used the WideResNet-28pre-trained feature backbones from Mangla et al.
; Yang et al
These experiments were conducted on three benchmarks, and each hyper-parameter configuration was averaged over 10, random trials.
Additional details and hyper-parameters are covered in the subsequent sections.
Additional Implementation Details for Sections 4.1 and 4.2 in the Main Paper:
The L2regularization coefficient was chosen for the mean-squared value of all classifier parameters, and the
Firth bias reduction coefficient was chosen for Equation (13) in the main paper.
Statistical Significance and Reducing the Effect of Randomized Factor: Our study investigates improvements over the baseline.
Random effects (random number seed; batch ordering; parameter initialization; and so on) complicate the study by creating variance in the measured improvement.
We used a matching procedure (so that the baseline and the Firth penalized models share the same values of all random effects) to control this variance.
As long as one does not search for random effects that yield a good improvement (we did not), this yields an unbiased estimate of the improvement.
Each experiment is repeated multiple times to obtain confidence intervals.
Note that (1) confidence intervals are small; and (2) improvements occur over a large range of feature backbones and datasets.
It is safe to conclude that Firth bias reduction reliably offers a small but useful improvement in accuracy for few-shot classifiers.
[Table] [Table] Additional Implementation Details for Section 4.3 in the Main Paper: We used two non-uniform count vectors with different average counts, 7 . 5 (scheme 1) and 15 (scheme 2), to generate the datasets in both validation and novel sets.
The count vector with the average of 7 . 5 -shots had 4 classes for each count from the geometric sequence { 2 , 4 , 8 , 16 } , and the count vector with the average of 15 -shots had 2 classes for each count from the arithmetic sequence { 1 , 5 , 9 , 13 , 17 , 21 , 25 , 29 } .
The same 1-layer logistic classifier of Sections 4.1 and 4.2 was used in Section 4.3 in the main paper.
Table A3 summarizes the hyper-parameters used in all the standard backbone experiments.
Also,
Figure A8 shows the relative accuracy improvements corresponding to Figure A7.
Figure A contains the validation accuracy versus Firth coefficient λ for the experiments of Figure 2 and
Figure A7.
Additional Implementation Details for Section 4.5 in the Main Paper:
In the experiments carried out in Section 4.5 in the main paper, a 1-layer cosine classifier was used.
Also, for the Firth bias-reduced cosine classifier, the regularization coefficient was tuned for each ( N, J ) pair, with
N representing the number of samples per class and J being the number of classes.
For J -way classification on the novel set when J happened to be larger than the number of classes in the validation set ( J val ), the coefficient tuned for J val -way
classification was adopted.
Resources: For the standard backbone experiments, we trained more than 384, 3-layer and 768,000 1-layer logistic classifiers for the balanced data settings.
For the imbalanced settings, we trained more than 64,000 3-layer and 128,000 1-layer logistic classifiers.
All classifiers [Table] were trained on CPUs, and these experiments alone consumed more than 16 CPU years.
Thebackbones were trained once using an Nvidia-V100 GPU and took 3 days for training.
For the cosine classifier experiments, we trained over 1.92, 1.92, and 3.36 million classifiers for mini-ImageNet, CIFAR-FS [Table] and tiered-ImageNet datasets, respectively.
These experiments used more than 5 CPU years in total.
E A DDITIONAL E XPERIMENTS ON THE S TANDARD B ACKBONES E. A DDITIONAL L OGISTIC C LASSIFIER E XPERIMENTS
The experiments of Figure 2 were repeated to perform 16-way classification using a logistic classifier on tiered-ImageNet and CIFAR-FS in Figure A13.
Moreover, 5-way and 10-way classification was tested for mini-ImageNet in Figure A6 in the same setting as Figure 2.
The results show that Firth improvements always exist and it is even more effective as the number of classes increases.
E. 3-L AYER L OGISTIC C LASSIFIERS FOR THE S TANDARD B ACKBONES
We conducted the same experiments as in Sections 4.1 and 4.2 in the main paper but with a 3layer logistic classifier.
Again, we see consistent accuracy improvements for the Firth bias-reduced classifier over the non-penalized (baseline) classifier in Figure A7.
This further supports the idea that Firth bias reduction boosts the performance of any reasonable classifier .
Needless to say,
L2-regularization is not effective as shown in Figure A7.
Furthermore, the imbalanced few-shot classification in Section 4.3 in the main paper was repeated with the 3-layer logistic classifier in Figure A10.
Again for both schemes, the Firth penalized classifier has larger accuracy improvement than the classifier penalized with the KL-divergence to the non-uniform prior over the class probabilities.
This further validates the effectiveness of Firth bias reduction in reducing the parameter estimation bias present in the few-shot setting.
F A DDITIONAL FEATURE BACKBONES
To test the Firth bias reduction technique for additional backbones, we used pre-trained DenseNet and MobileNet backbones on tiered-ImageNet from Wang et al.
The accuracy improvements
[Table]
of Firth
penalized logistic classifier over the baseline averaged over 10,000 trials are plotted in
Figure A11.
Regardless of the number of classes, the improvements are always positive.
G C OMPARING F IRTH B IAS R EDUCTION A GAINST S TANDARD L ABEL S MOOTHING T ECHNIQUES
To demonstrate that Firth bias reduction cannot simply be replaced with label smoothing, we tested two advanced variants of label smoothing that are superior to the original version as proposed by Pereyra et al.
The first variant, called confidence penalty, uses the entropy of the classifier’s output (or equivalently, reverses the direction of the KL divergence in the original version of label smoothing (Szegedy et al., 2016)); and the second variant, called unigram label smoothing, uses prior distribution over the classes instead of uniform, which has been shown to be advantageous when the output labels’ distribution is imbalanced in Pereyra et al.
Note that both variants were investigated for training a full deep neural network with a feature extractor backbone in large-sample regimes in Pereyra et al.
Our experiments in Figure 3 evaluate the effect of unigram label smoothing when training the classifier in the small-sample regime.
We also performed more experiments in the same setting to compare Firth bias reduction against confidence penalty regularization.
As summarized in Table A4, in all the settings Firth bias reduction has larger significant improvements than the confidence penalty technique.
This further supports the value of using Firth bias reduction and the fact that its impact on few-shot classification cannot be reproduced with well-known and widely-used label smoothing techniques. [Table]
H A DDITIONAL C OMPARISON WITH S TATE OF THE A RT
Table A5 summarizes the accuracy improvements obtained by integrating Firth bias reduction with the distribution calibration method (Yang et al., 2021) under different shots and ways on the tieredImageNet dataset.
This method calibrates the features to follow a normal distribution [Table] and generates artificial samples from the estimated normal distribution as data augmentation to aid few-shot classification.
In its state-of-the-art setting, the features are transformed using Tukey transformations, 750 artificial samples are generated per class, and a logistic classifier is used.
We tested Firth bias reduction in two scenarios: (1) state-of-the-art setting without generating artificial samples from the calibrated distribution; and (2) state-of-the-art setting with 750 artificial samples generated per class, [Figure] li [Figure]
i
[Figure]
n [Figure] cc u r a c [Figure]
Logistic
[Figure]
a
[Figure]
d a t i o
[Figure]
A cc u r a c
[Figure]
li d a t i o n
A cc u r a c y
[Figure]
as shown in Table A5.
The 95% confidence intervals for the accuracy improvements are reported in both
As shown in Table A5, Firth bias reduction produces positive improvements in all cases, which are statistically significant in all the cases.
As expected, Firth bias reduction produces larger improvements when artificial sample generation is disabled, and thus there is a larger maximum likelihood estimation bias on the logistic classifier (despite the use of Tukey transformation).
In the case that artificial samples are added and they are tuned to follow the original normal distribution, having Number of Shots [Figure] Firth Bias Reduction (tiered-ImageNet)
ResNet
ResNet
ResNet ResNet ResNet Number of Shots     (
[Figure]
L2 Regularization (tiered-ImageNet)
ResNet
ResNet
ResNet
ResNet
ResNet
Number of Shots
[Figure]
[Figure]
Reduction (CIFAR-FS)
ResNet
ResNet
ResNet
ResNet
ResNet
Number of Shots
[Figure]
L2 Regularization (CIFAR-FS)
ResNet
ResNet
ResNet
ResNet
ResNet of them can to some extent alleviate the bias in the estimation of the logistic classifier’s weights in the few-shot regime.
However, the results show that even in the presence of more data (artificial samples), Firth bias reduction is still effective.
This suggests that producing artificial samples to augment data cannot resolve the estimation bias issue, as they are more likely to be similar to the limited real samples available.
As shown in Section 4.6, this becomes an even more severe problem in the cross-domain few-shot setting; producing artificial samples is significantly less effective than
Firth bias reduction, due to the domain shift.