O N THE I MPORTANCE OF F IRTH B IAS R EDUCTION IN F EW -S HOT C LASSIFICATION
A BSTRACT
Learning accurate classiﬁers for novel categories from very few examples, known as few-shot image classiﬁcation, is a challenging task in statistical machine learning and computer vision.
The performance in few-shot classiﬁcation suffers from the bias in the estimation of classiﬁer parameters; however, an effective underlying bias reduction technique that could alleviate this issue in training few-shot classiﬁers has been overlooked.
In this work, we demonstrate the effectiveness of Firth bias reduction in few-shot classiﬁcation.
Theoretically, Firth bias reduction removes the O ( N − 1 ) ﬁrst order term from the small-sample bias of the Maximum Likelihood Estimator.
Here we show that the general Firth bias reduction technique simpliﬁes to encouraging uniform class assignment probabilities for multinomial logistic classiﬁcation, and almost has the same effect in cosine classiﬁers.
We derive an easy-to-implement optimization objective for Firth penalized multinomial logistic and cosine classiﬁers, which is equivalent to penalizing the cross-entropy loss with a KL-divergence between the predictions and the uniform label distribution. Then, we empirically evaluate that it is consistently effective across the board for few-shot image classiﬁcation, regardless of (1) the feature representations from different backbones, (2) the number of samples per class, and (3) the number of classes.
Finally, we show the robustness of Firth bias reduction, in the case of imbalanced data distribution.
Our implementation is available at https://github.com/ehsansaleh/ﬁrth_bias_reduction.
I NTRODUCTION
Few-shot image classiﬁcation is the practice of learning accurate classiﬁers using a small number of labeled samples (Fei-Fei et al., 2006; Vinyals et al., 2016; Wang and Hebert, 2016;
Finn et al., 2017; Snell et al., 2017; Wang et al., 2020).
It has a wide range of applications from face and gesture recognition (Pﬁster et al., 2014) to visual navigation in robotics (Finn et al., 2017).
Essentially, modern few-shot classiﬁcation methods can be viewed as a combination of learning (1) a strong feature representation through a backbone network (e.g., Verma et al.
; Gidaris et al. Tian et al.
(2020)), and (2) an accurate small-sample classiﬁer (e.g.
, Wang and Hebert (2016); Chen et al.
Therefore, two key questions arise in few-shot image classiﬁcation: (1) How can we obtain a strong feature representation?
and (2) How can we train accurate classiﬁers using a small number of samples?
There have been many existing methods addressing the former question using a host of different techniques (Snell et al., 2017; Rusu et al., 2019; Verma et al., 2019; Mangla et al., 2020).
Here we focus on the less-explored second question.
For our purposes, we will use standard feature representations and methods for training the backbone model for few-shot classiﬁcation.
That still leaves us with a severe classiﬁer problem than most people realize.
There is a substantial difﬁculty with training a classiﬁer using a small number of samples.
In particular, with very few samples, standard classiﬁcation machinery is biased.
In other words, although the Maximum Likelihood Estimators (MLEs) are statistically consistent and asymptotically normal (Fahrmeir and Kaufmann, 1985), it is well-established that MLEs are biased for a small number of N samples, with bias of O ( N − 1 )
(Cox and Snell, 1968; Box, 1971; Whitehead, 1986; Firth, 1993).
Since common logistic regression models are a type of MLEs, they are also biased (Schaefer, ; Cordeiro and McCullagh, 1991; Firth, 1993; Steyerberg et al., 1999).
Such biases increase the error rate of the few-shot trained classiﬁers, and so are important in few-shot learning.
In fact, there is a standard solution for bias prevention by modifying the ordinary MLEs – known as Firth’s Penalized Maximum Likelihood Estimator (PMLE) (Firth, 1993).
In the case of the exponential family of distributions, Firth has a simpliﬁed form that penalizes the likelihood by
Jeffrey’s invariant prior (Firth, 1993; Poirier, 1994), which is proportional to the determinant of the
Fisher Information Matrix F .
For logistic and cosine classiﬁers (Chen et al., 2019) which are widely used in few-shot classiﬁcation, since they belong to the exponential family, Firth bias reduction can be further cast as adding a log-determinant penalty ( log(det( F )) ) to the cross-entropy loss.
While such standard strategies can control the bias in classiﬁers trained with very few samples, they have not been utilized in few-shot image classiﬁcation tasks.
In this paper, we show that using Firth bias reduction produces reliable improvements in a wide range of circumstances .
We achieve this by deriving a simpliﬁed yet effective Firth formulation that penalizes the KL-divergence between the predicted distribution and the uniform distribution of classes, for both multinomial logistic regression models and cosine classiﬁers.
Note that common regularization techniques (such as L2-regularization and label smoothing (Szegedy et al., 2016)) cannot reduce the estimation bias of classiﬁer weights in small-sample regimes (Liu et al., 2020) as
Firth’s penalty does; these regularization techniques are mainly used to control model complexity of deep neural networks for training feature extractor backbones in large-sample regimes .
More concretely, our results indicate that the improvements produced by Firth bias reduction for few-shot image classiﬁcation tasks are consistent across the board (1) on a wide range of feature representations, (2) with both balanced and imbalanced data, (3) for both single-layer and multi-layer classiﬁers, (4) for both logistic and cosine classiﬁers, and (5) over multiple datasets and problems.
Importantly, we found Firth bias reduction to consistently yield statistically signiﬁcant and positive improvements , and we did not observe any performance penalty for utilizing it.
Such improvements are on the order of 0.5-2.5% and up to 3% in challenging tasks with large number of classes.
Our main contributions include (1) deriving a generalized expression for Firth bias reduction in fewshot multinomial logistic regression models, and providing geometrical insight into its effect on the classiﬁcation probability space; (2) evaluating the efﬁcacy of the Firth penalized multinomial logistic model in few-shot scenarios, both with balanced and imbalanced data distributions; (3) showing that Firth bias reduction can be extended beyond typical logistic models, and can be successfully adopted in cosine classiﬁers; and (4) providing an empirical comparison of Firth bias reduction with common regularizers such as L2 and label smoothing.
B ACKGROUND
Mathematical Notations:
In this work, we assume a multinomial logistic regression model for the classiﬁer, with a total of J + 1 classes { 0 , 1 , 2 , · · · , J } .
The logistic regression weights for class j is denoted as β j ( 1 ≤ j ≤ J ).
The class j = 0 is the base class with zero logistic regression weights.
We assume to have a total number of N samples D
= { ( x 1 , y 1 ) , · · · , ( x N , y N ) } .
The i th target y
i is the one-hot encoding of the i th label.
The assignment probability of the i th sample to class j is denoted as P i,j :
[Equation]
The likelihood of the sample set D given the weights β is denoted as Pr( y | x ; β ) :
[Equation]
where 1 [ · ] denotes the binary indicator function.
We may abuse the notation and use Pr( y | x ) instead of Pr( y | x ; β ) in the derivation processes.
Therefore, the logistic log-likelihood function L logistic is deﬁned as
[Equation]
[Figure]
The Fisher Information Matrix (FIM) is deﬁned as the Hessian of the negative log-likelihood function
L logistic :
[Equation]
The Maximum Likelihood Estimator (MLE) for logistic regression is deﬁned as
[Equation]
The dimension of the feature space is denoted as d in the derivations.
U [ a,b ] denotes the (discrete) uniform distribution in the [ a, b ] interval.
The cross-entropy and the KL-divergence of distributions p, q are deﬁned as
[Equation]
Table A2 in the Appendix summarizes these notations.
Small-Sample Bias of MLE:
Assume β ∗ is the true generative parameter.
When the sample size N is small, it is shown that the MLE bias b (ˆ β MLE ) :=
E [ˆ β MLE − β ∗ ] is non-zero and of O ( N − 1 )
(Cox and Snell, 1968).
Therefore, while MLE is unbiased as N → ∞ , it is inaccurate for few-shot learning.
Firth Bias Reduction for MLE:
Firth’s Penalized Maximum Likelihood Estimator (PMLE) (Firth, 1993) is a modiﬁcation to the ordinary MLE, which removes the O ( N − 1 ) term from the small-sample bias.
In particular, Firth has a simpliﬁed form for the exponential family .
When Pr( y | x ; β ) belongs to the exponential family of distributions, the effect is to penalize the likelihood by Jeffrey’s invariant prior (Poirier, 1994), which is proportional to the determinant of F ( det( F ) ).
Logistic and cosine classiﬁers (Chen et al., 2019) – the widely-used classiﬁcation models in few-shot learning – belong to the exponential family; so for them, Firth bias reduction simpliﬁes to adding a penalty ( log(det( F )) ) to the cross-entropy loss.
In what follows, we will derive a further simpliﬁed yet effective Firth formulation for logistic and cosine classiﬁers, which is computationally more efﬁcient, deals with the case when det( F ) = 0 , and generalizes to multinomial distributions.
MLE vs. Firth’s PMLE for a Simple Case: To demonstrate the extent of the MLE bias and how Firth’s PMLE removes the leading O ( N − 1 ) bias term, we simulated data from the geometric distribution with probability of success β = 0 .
5 as its only parameter.
The geometric experiment was chosen since a closed-form solution for the MLE and Firth’s PMLE can be derived. Given the samples ( y 1 , · · · , y N ) from the geometric distribution, the sample mean is ¯ y = 1 N (cid:80)
y
i and we have ˆ β MLE = 1 ¯ y and ˆ β Firth = N − 1 N ¯ y − 1 .
Note that since the sample mean is noisy, the MLE suffers from a noisy denominator, making it biased.
Figure 1 shows that ˆ β
Firth is close to the true parameter β ∗ for all sample sizes, whereas ˆ β MLE has a signiﬁcant bias away from β ∗ for small N .
To further validate that the MLE bias is indeed of O ( N − 1 ) , we plotted the MLE bias against the sample size in the log-log scale in Figure1, which shows that it is closely following a line with a negative unit slope.
F IRTH B IAS R EDUCTION IN L OGISTIC AND C OSINE C LASSIFIERS
In logistic models, the penalized likelihood function proposed by Firth is equivalent to imposing
Jeffreys’ prior (Poirier, 1994) on the parameters and making a maximum a posteriori estimation.
In particular, Firth bias reduction encourages models with “ large ” F by multiplying the likelihood by det( F ) .
This penalty degenerates when det( F ) = 0 .
We work in the highest dimensional subspace where F has full rank, and use det( F | r ) to denote the product of all r non-zero eigenvalues of F , obtaining
[Equation]
(7) where Z is a normalization constant and det( F | r ) 1 / 2 is the Jeffery’s prior.
Taking the log of both sides yields the log-posterior as a sum of the logistic log-likelihood function and the Firth bias reduction term:
[Equation]
8) where we have
[Equation]
The deﬁnition of L Firth was left ambiguous up to a constant with respect to β to facilitate the Firth bias reduction term’s interpretation and avoid the deﬁnition of similar terms.
Furthermore, λ controls for the impact of the Firth term on the outcome relative to L logistic .
We then apply a series of derivation steps to simplify Equation (9), which are left to Section A in the Appendix.
Finally, the Firth bias reduction term can be expressed as
[Equation]
For the cosine classiﬁer, the proof that log(det( F )) simpliﬁes to Equation (10) involves straightforward manipulation of the proof for the logistic classiﬁer, and is left to Section B in the Appendix.
The normalization of the β j weights in the cosine classiﬁer turns into a pure scale term in the optimization, and for a cosine classiﬁer, scaling of the β j weights does not affect predictions.
Therefore, this term should be ignored, and the bias reduction term effectively becomes the same as Equation (10).
Interpreting the Firth Bias Reduction for Logistic Models: It is well known that Jeffery’s prior shrinks the parameter estimates towards zero, which is equivalent to encouraging uniform class assignment probabilities (Firth, 1993;
Bull et al., 2002).
We take an alternative approach to reach the same conclusion in the following.
By re-arranging Equation (10), one can see the (cid:80) Jj =0 log( P i,j ) term as a scaled average over a uniform distribution of classes:
[Equation]
Therefore, we can abuse the notation, and redeﬁne the coefﬁcient λ and the constant to have
[Equation]
This means that by dropping the constants, the optimization objective L can be rewritten as
[Equation]
Since the KL-divergence and the FIM deﬁne an information geometry and a Riemanian metric on probabilistic measure spaces (Nielsen, 2020), a geometrical insight into Equation (13) is provided in
Figure A8 in the Appendix as well.
Firth Bias Reduction vs. Common Regularization: While this insight brings Firth bias reduction closer to the common regularization techniques (e.g., L2-regularization), it is worth noting that (1) common regularization techniques mainly focus on controling model complexity, instead of reducing small-sample estimation bias; (2) Firth bias reduction operates on a much lower-dimensional target distribution space, unlike L2 which operates in the high-dimensional parameter space; (3) Firth uses the same kind of metric as the logistic loss; and (4) Firth bias reduction is dimensionally consistent like the Natural Gradients (Amari, 1998; Pascanu and Bengio, 2013), whereas L2 is not.
Firth Bias Reduction vs. Label Smoothing: Notice that the original form of label smoothing used for training in large-sample regimes (Szegedy et al., 2016) has the same formulation as the simpliﬁed
Firth penalty term in Equation (12) for multinomial logistic classiﬁers .
However, Firth bias reduction is inherently different in that it reduces the classiﬁer estimation bias in the small-sample regimes, whereas label smoothing penalizes over-conﬁdent predictions when training deep neural networks with large amounts of samples.
Generally, the Firth bias reduction term (i.e., log(det( F )) ) is not the same as the label smoothing penalty (i.e., D KL ( U (cid:107) P i ) ) for deep neural networks.
Additional analysis and empirical comparisons are provided in Section 4.4 and Section F in the Appendix.
E XPERIMENTAL R ESULTS
Here we show that for a wide range of experiments, Firth bias reduction is a reliable source of small yet useful improvements in the performance.
Since we report improvements for a wide range of methods and settings, the absolute accuracy improvements were reported.
These absolute improvements sometimes constitute signiﬁcantly to the baseline accuracy in terms of relative importance.
Datasets: We perform experiments on four widely-used and publicly available benchmarks: miniImageNet (Vinyals et al., 2016), CIFAR-FS (Bertinetto et al., 2019), tiered-ImageNet (Ren et al., 2018), and CUB (Welinder et al., 2010).
Each dataset consists of non-overlapping base, validation, and novel classes.
The detailed class splits are described in Section D in the Appendix.
Following the standard practice (Chen et al., 2019), we train feature backbones on base classes, cross-validate bias reduction coefﬁcients on validation classes, and train classiﬁers and measure test accuracy over multiple trials on novel classes.
Implementation Details: Details regarding the setup, implementation, statistical signiﬁcance, and reducing the effect of randomized factors are covered in Appendix Section D.
Baselines and Evaluation Metric: Non-penalized classiﬁers are used as the baseline in all experiments to compare Firth bias reduction and L2-regularization.
Absolute accuracy improvements over the baseline averaged across multiple trials are used as the evaluation metric in all experiments.
Relative improvements are also shown in the Appendix, which demonstrate similar behaviors.
Summary of Results: Section 4.1 shows the efﬁcacy of Firth bias reduction on standard feature backbones (ResNets with varying depth) and single-layer logistic classiﬁers.
Section 4.2 shows and argues that Firth bias reduction outperforms L2-regularization on few-shot classiﬁcation tasks.
Next, we investigate the driving factor in Firth’s improvement in Section 4.3, and show evidence for the efﬁcacy of Firth’s bias suppression property.
We also show that Firth bias reduction can be effectively, and without any modiﬁcations, applied to imbalanced data distribution settings.
Section 4.4 compares Firth bias reduction against label smoothing variants, and shows that Firth outperforms label smoothing.
In Section 4.5 Firth bias reduction is applied to modern few-shot methods with advanced feature backbones (i.e., WideResNet trained with strong regularization (Mangla et al., 2020)) and cosine classiﬁers.
Experiments with additional feature backbones (DenseNet and MobileNet (Wang et al., 2019)) are included in
Appendix Section G. Finally, Section 4.6 demonstrates that Firth bias reduction produces reliable improvements over the state-of-the-art feature calibration method (Yang et al., 2021).
Our collective results clearly indicate a consistent pattern of improvements over a large array of (1) feature representations, (2) datasets, (3) classiﬁcation ways, (4) number of shots, (5) types of classiﬁers, and (6) with both balanced and imbalanced data distributions. Number of Shots 0. [Figure]
Firth Bias Reduction ResNet ResNet ResNet
ResNet
ResNet
Number of Shots 0.
( A
CC ) %
L2 Regularization
ResNet
ResNet
ResNet
ResNet
ResNet
4. F IRTH B IAS R EDUCTION I MPROVES THE A CCURACY OF L OGISTIC C LASSIFIERS
Figure 2 summarizes the results.
For all the k -shot tasks and backbones, the average absolute test accuracy improvement is signiﬁcant and it increases as k increases.
Also, Firth bias reduction does not hurt the 1 and 5 -shot performance, with slight improvements for some backbones.
Extra experiments are included in Appendix Section E.1.
4. L2-R EGULARIZATION IS NOT AS E FFECTIVE AS F IRTH B IAS R EDUCTION
To explore whether the performance improvement pattern for Firth bias reduction can be reproduced by common regularization techniques such as L2, the same experiments were repeated with L2regularized logistic classiﬁers.
For almost all the k -shot tasks and feature backbones, the average absolute test accuracy improvement is close to zero, as shown in Figure 2.
This suggests that the bias reduction property of Firth plays a signiﬁcant part in improving the classiﬁer’s performance in the few-shot regime which cannot be achieved by L2- regularization.
4. B IAS OR P RIOR ?
Firth bias reduction clearly helps.
It may be doing so because it is an effective way of suppressing bias.
Alternatively, reinforcing the prior information might be helping.
This section offers evidence that bias suppression is what is important.
We consider data where the class frequencies are imbalanced.
We apply a variant of Firth bias reduction that uses class prior frequencies to this data.
This variant is not as successful as routine Firth bias reduction, suggesting that bias correction is what is important.
One might consider replacing the uniform distribution in the Firth prior with a class probability distribution.
That is, replacing L Firth = − λ · D KL ( U [0 ,J ] (cid:107) P i ) by L Firth = − λ · D KL ( A (cid:107) P i ) with A being the imbalanced class distribution, when training a Firth penalized few-shot logistic classiﬁer.
To test this, we designed two schemes to generate imbalanced datasets.
These schemes differ in the imbalanced count vectors used to create the few-shot dataset in validation and novel splits.
The increments in the counts per class were designed to result in two different average counts of 7.
(Scheme 1), and 15 (Scheme 2) over the classes (experimental details are left to Section D in the
Appendix).
For each scheme, the experiments were carried out similar to the “balanced few-shot” case on mini-ImageNet, except that in each trial three classiﬁers – (1) baseline (not penalized), (2) Firth penalized (KL with the uniform prior), and (3) modiﬁed Firth penalized (KL with the imbalanced
R N
[Figure]
R N R N
R N R N
[Figure]
7.5 Average Shots
[Figure]
R N
R N R N
R N R N
[Figure]
CC [Figure]
[Figure]
The difference of accuracy improvements between the uniform and the non-uniform priors is more striking for the 7. case, which suffers from a greater deal of estimation bias than the 15-average-shot scenario. This shows that the improvements of Firth bias reduction are driven by its bias suppression property, and are not a mere consequence of imposing a uniform class prior. The same experiments were repeated with a 3-layer classiﬁer in Figure A10 in the Appendix prior) – were trained.
In both schemes, the improvement achieved by the Firth penalized is higher than that of the modiﬁed Firth penalized classiﬁer over the baseline for all backbones, as shown in Figure 3.
The difference between the two improvements is more substantial in Scheme 1, where the average of samples per class is less than Scheme 2.
This implies that the Firth penalized classiﬁer is indeed reducing the bias and should not be naively considered as if a prior is simply deﬁned over the class assignment probabilities.
4. F IRTH B IAS R EDUCTION I MPROVES O VER L ABEL S MOOTHING
Even though the original version of label smoothing (Szegedy et al., 2016) has the same formulation as the Firth bias reduction term in Equation (13), there is a family of label smoothing techniques that aim to reduce overﬁtting and over-conﬁdent predictions (Pereyra et al., 2017).
Namely, the conﬁdence penalty (i.e., regularizing D KL ( P i (cid:107) U ) ) and unigram label smoothing (i.e., regluarizing the
KL-divergence with class priors) are two label smoothing variants, which are shown to outperform the original label smoothing, when training complex networks with large amounts of samples (Pereyra et al., 2017).
Figure 3 shows that Firth bias reduction outperforms unigram label smoothing, and
Table A4 in the Appendix shows that Firth bias reduction yields better results than applying conﬁdence penalties and entropy regularization.
This further suggests that the improvements obtained by Firth are the result of its bias reduction property.
4. F IRTH B IAS R EDUCTION I MPROVES THE P ERFORMANCE OF C OSINE C LASSIFIERS
As cosine classiﬁers have recently proven to be useful for few-shot classiﬁcation (Gidaris and
Komodakis, 2018; Chen et al., 2019), we investigated the impact of Firth bias reduction on them.
The difference between a logistic and cosine classiﬁer parameter space is that the parameter space of the cosine classiﬁer is constrained to a unit sphere, therefore it is likely for the Firth bias reduction to improve cosine classiﬁer performance as well.
To test this, we based our experiments on the implementation of the S2M2 R method (Mangla et al., 2020), and used their pre-trained WideResNet28-10 (Zagoruyko and Komodakis, 2016) on the base classes as a strong feature extractor.
We trained
Firth penalized and non-penalized cosine classiﬁers on the mini-ImageNet, tiered-ImageNet, and
CIFAR-FS datasets.
Few-shot classiﬁers were trained for varying number of classes, depending on [Figure] Number of Classes [Figure]
[Figure] [Figure] [Figure]
[Figure] CIFAR-FS 1-shot 5-shot 10-shot [Figure] the dataset, and varying number of samples per class.
We followed the same scheme as Mangla et al. (2020) to generate the k -shot support and query datasets for training and testing, respectively. As shown in Figure 4, in all datasets, Firth bias reduction improves the performance of 5-, and 10-shot tasks in a monotonically increasing fashion with respect to the number of classes.
This behavior is well represented for tiered-ImageNet, as increasing 5-way to 150-way classiﬁcation results in more than 2.5% improvement of the average test accuracy.
In the 1 -shot task, the improvement by Firth bias reduction is monotonically decreasing with the number of classes in all the datasets except tieredImageNet, but it does not hurt the performance.
The improvement almost falls within (0.1%-0.5%), (0-0.1%), and (0-0.2%), for mini-ImageNet, CIFAR-FS, and tiered-ImageNet, respectively.
This experiment suggests that not only is Firth bias reduction effective for cosine classiﬁers, but it also could help with the performance even when strong features are used.
This is well justiﬁed as
Firth bias reduction targets the bias introduced in the classiﬁer parameters not the features.
4. C OMPARISON WITH A DDITIONAL S TATE - OF - THE -A RT M ETHOD
We further applied the Firth bias reduction term to the recent state-of-the-art method in few-shot classiﬁcation (Yang et al., 2021), called Distribution Calibration (DC).
DC computes the closest base classes to each sample in the feature space, and samples artiﬁcial examples from a Gaussian distribution centered around the mean of the closest base class features.
To make the setup challenging, we consider data where the base classes are categorically different from the novel classes: i.e., by performing cross-dataset few-shot classiﬁcation, where the features are trained on the base classes of mini-ImageNet and tiered-ImageNet, and the novel sets are instead sampled from the CUB dataset.
The results are shown in Table 1, and indicate that the Firth bias reduction improves over Yang et al.
We also show the Firth bias reduction improvements on the tiered-ImageNet dataset in Appendix Section H and Table A5.
This further supports the observation that Firth bias reduction yields small but reliable improvements under different methods.
[Table]
R ELATED W ORK
Bias Reduction of the MLE:
A myriad of statistical work has been proposed to mitigate the smallsample bias of the MLE under different settings (Anderson and Richardson, 1979; Kenward and
Roger, 1997; Bull et al., 2002; Kosmidis and Firth, 2009).
Originally, the asymptotic bias of MLE was shown to be of O ( N − 1 ) , with N being the sample size (Firth, 1993).
To counter such an estimation bias, many approaches have existed.
To name a few, (1) additive penalization terms to the main logistic loss were proposed to reduce the MLE’s bias (Firth, 1993;
Bull et al., 2002; Greenland and
Mansournia, 2015), and (2) some methods have been proposed to directly approximate and remove such a small sample bias (Cox and Hinkley, 1979).
While the latter approach may sound appealing, estimating the MLE’s bias can be impractical.
For instance, in few-shot scenarios, a perfect separation of the classes may be achievable, causing the logistic MLE to be unbounded (Heinze and Schemper, 2002).
On the other hand, the penalization methods do not modify the estimated parameters directly, and instead gently push for a preference towards less biased estimates.
Such penalization methods can be generally applicable to a vast array of models.
Theoretically, Firth’s PMLE reduces the bias by removing the leading O ( N − 1 ) term from the MLE’s bias (Firth, 1993) – a property that does not exist in common regularization techniques such as L2-regularization.
Furthermore, PMLE of the logistic model has been shown to have smaller variance than MLE as well (Copas, 1988; Kosmidis and Firth, 2009).
Firth’s PMLE has been well studied for binomial logistic regression (Firth, 1993), and applied and tested against other penalization techniques in other ﬁelds (Rainey and McCaskey, 2015; Muchlinski et al., 2016; Rahman and Sultana, 2017).
Additional related work on few-shot image classiﬁcation was left to Appendix Section C.
C ONCLUSION
We show that Firth bias reduction consistently improves the accuracy across the board in few-shot classiﬁcation regardless of (1) the employed feature backbone, (2) the number of classes and samples, and (3) the dataset and problem setting.
Furthermore, our experiments show that Firth bias reduction can improve the performance of the cosine classiﬁers, and is applicable to imbalanced few-shot settings without any necessary modiﬁcations.
Overall, our evaluations suggest that Firth bias reduction is a useful and general bias reduction tool that has been missing in few-shot classiﬁcation, and should be incorporated in few-shot classiﬁcation tasks for accuracy improvements.  A CKNOWLEDGMENT
This work utilized resources supported by (1) the National Science Foundation’s Major Research
Instrumentation program (Kindratenko et al., 2020), grant number and (2) the National
Science Foundation’s Creating Knowledge with All-Novel-Class Computer Vision program, grant number 2106825.
Also, this work made use of the Illinois Campus Cluster, a computing resource that is operated by the Illinois Campus Cluster Program (ICCP) in conjunction with the National Center for Supercomputing Applications (NCSA) and which is supported by funds from the University of
Illinois at Urbana-Champaign A F IRTH B IAS R EDUCTION FOR F EW -S HOT M ULTINOMIAL L OGISTIC R EGRESSION
Table A2 summarizes the notations used throughout the main paper and here.
Given a dataset of N samples D = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , · · · , ( x N , y N ) } , the multinomial logistic model for a total of J +classes can be formulated as
[Equation]
where class j = 0 was chosen as the base class in the log odds ratio.
In other words, w.l.o.g. we assume β 0 = 0 in this formulation.
Given the decision rule in Equation (A14), we can write
P i,
[Equation]
Under this notation, the log-likelihood would be L logistic =
(cid:80) Ni =1 (cid:80) Jj =
[Equation]
1 [ y i =
j ] · log( P i,j ) .
The data matrix X D is given as
[Equation]
Also, X := X D ⊗ I J where the ⊗ operator denotes the Kronecker matrix product, and the Jdimensional identity matrix is denoted as I J .
Firth (1993) has established that the bias of logistic regression can be removed by maximizing the sum of (1) the log-likelihood L logistic and (2) the log-determinant of the Fisher Information Matrix (FIM).
For our purposes, this presents some challenges: we have a few number of samples and the
FIM determinant is zero.
Instead, we use the product of all non-zero eigenvalues of the FIM as its “ amended determinant ”.
To obtain this efﬁciently, we need to know the speciﬁc structure of the FIM.
It is important in what follows that the FIM can be deﬁned as the matrix product
[Equation]
where M is a block-diagonal matrix whose i th diagonal block is denoted as M i .
We leave the deﬁnition of X and M i to the “ FIM Formulation for Logistic Regression ” subsection.
Next, we focus on:
• Determinant Amendment and Constant Dropping : Generically, we show that
[Equation]
where det( F | NJ ) is an amended version of det( F ) .
• Efﬁcient Computation of log(det( M i ))
: Next, we show that
[Equation]
Combining these two points will lead us to the simpliﬁed Firth bias reduction objective:
[Equation]
Determinant Amendment and Constant Dropping: Having F = X (cid:124)
· M · X prompts us to utilize the SVD of X as
[Equation]
Therefore, the FIM can be written as
[Equation]
where K := U (cid:124) · M · U .
Since V and U are rotation matrices, we can write det( F ) = det( S (cid:124) · K · S ) , and det( K ) = det( M ) =
N (cid:89) i = det( M i ) .
(A22)
[Table]
The Logistic Regression Weights for Class j ( j ∈ { 1 , 2 , · · · , J } )
[Table]
Classiﬁcation Probability of Sample i
Belonging to Class j
[Table]
The i th Sample’s Soft Classiﬁcation Probabilities
[Table]
Regression Sample Dataset
[Table]
The One-Hot Encoding of the Label y
i
[Table]
The Binary Indicator Function (i.e., 1 when a = b and 0 otherwise)
[Table]
The Fisher Information Matrix
[Table]
the Features
[Table]
The J × J Identity Matrix
[Table]
The Kronecker Product of Matrix A by Matrix B
[Table]
The All Ones Matrix with r Rows and c Columns
[Table]
The Maximum Likelihood Estimator (MLE)
[Table]
The Logistic Log-Likelihood Function
[Table]
The Firth Bias Reduction Function
[Table]
The Firth Bias Reduction Coefﬁcient
CE ( p (cid:107) q )
The Cross-Entropy of p and q
D KL ( p (cid:107) q )
The KL Divergence of p and q
U [0 ,J ]
The Uniform Class Assignment Probabilities det( A | r )
The Amended Determinant of the Degenerate Matrix A with at most r
Non-zero Eigenvalues (See Section A of the Appendix)
As we have d > N for most few-shot tasks, the matrix S can be viewed in the following form:
[Equation]
(cid:105)
[Equation]
(A23) where ˆ S is a diagonal square matrix.
Since ˆ S (cid:124)
=
ˆ S , we have
S (cid:124)
[Equation]
 (cid:18)
[Equation]
NJ × NJ [Equation] (A24) Equation (A24) and det( F ) = det( S (cid:124) · K · S ) show why det( F ) is zero.
For mitigation, we replace det( F ) with the product of the non-zero eigenvalues of F , namely det( F | NJ ) , and call it the “ amended determinant ” of F .
Thanks to F = V · (cid:0) S (cid:124) · K · S (cid:1) · V (cid:124) , even the amended determinants det( F | NJ ) and det( S (cid:124) · K · S | NJ ) are the same: [Equation] A
[Equation]
Number of Shots
[Figure]
Firth Bias Reduction
ResNet
ResNet ResNet
ResNet ResNet
Number of Shots 0.
[Figure]
L2 Regularization
ResNet ResNet
ResNet ResNet
ResNet
Could det( F | NJ ) be zero?
The answer is “ not ” generically; det( M ) is generically positive as we will show det( M i )
= (cid:81)
Jj =
P i,j later in Equation (A28).
Also, det( ˆ S 2 ) > 0 holds with probability 1 for continuous data distributions.
In fact, det( ˆ S 2 ) can only be zero when the data contains linearly dependant samples, which happens with zero probability for non-atomic data distributions.
Therefore, we have
[Equation]
This is the same as Equation (A18): since the log(det( ˆ S 2 )) term is independent of the model’s parameters, we can treat it as an optimization constant and drop it.
The FIM Formulation for Logistic Regression: Elementary methods established that the FIM of the logistic classiﬁer is composed of J × J block matrices, each with a dimension of d × d (where d is the dimension of the features).
These block matrices can be expressed as
[Equation]
This facilitates the expression of the FIM as shown in Equation (A17): We can deﬁne X as the
Kronecker product of (1) the data matrix X D and (2) the J -dimensional identity matrix I J :
[Equation] M would then be a block-diagonal matrix whose i th block is deﬁned as [Equation] [Figure]
[Figure]
Efﬁcient Computation of log(det( M i )) :
We deﬁne the soft predictions of the i th sample (excluding the base class) as P i, 1: J := (cid:104) P i, [Equation] (cid:124) .
Given M
i ’s deﬁnition in Equation (A29), we can write
[Equation]
Next, we use the Matrix-Determinant Lemma (David, 1998) to compute det( M i ) : det( M i ) = det( Diag ( P i
[Equation]
[Equation] J ))
[Equation]
i
[Equation]
[Equation]
Taking the log will give us Equation (A19).
B F IRTH BIAS REDUCTION FOR C OSINE C LASSIFIERS
Section A derived the Firth bias reduction term for the logistic classiﬁer as D KL ( U (cid:107) P i ) .
Here, we generalize this observation to cosine classiﬁers, and prove that the Firth bias reduction for cosine classiﬁers reduces down to the same form as the one obtained for logistic classiﬁers.
By deﬁning the normalization transformation
[Equation]
we have the log-likelihood relation between the cosine classiﬁer and the logistic model:
[Equation]
According to Equation (4) of the main paper and the chain rule, we can write
[Equation]
where A d ( J +1) × d ( J +1) is the Jacobian matrix of T ( β ) with respect to β .
It can be shown that A is a symmetric block-diagonal matrix, whose j th diagonal block is
[Equation]
Therefore, log(
[Equation]
[Figure]
It is obvious that
[Equation]
Therefore, A j,j can be thought as an identity-proportional matrix in the sub-space perpendicular to β j .
Therefore, its amended determinant is
[Equation]
For the plain logistic model, the Firth penalized optimization problem is:
[Equation]
For the cosine classiﬁers, the Firth penalized optimization problem is:
[Equation]
) where we have C := 2 λ ( d − 1) .
By re-parameterizing β = ( u β , s β ) with u β := T ( β ) and s β := (cid:2) (cid:107) β 0 (cid:107) , · · · , (cid:107) β J (cid:107) (cid:3) , we have
[Equation]
Since u β and s β are independent optimization parameters, this problem can be restated as
[Equation]
Since s β has no effect on the predictions of the model, it can be ignored.
Therefore, we end up with the optimization problem
[Equation]
K L K L Essentially, this suggests applying the same bias reduction form to cosine classiﬁers as the one for logistic log(det( F logistic ( u β , T ( x ) , y ))) reduces down to D KL ( U (cid:107) P i ) as proven
Section the cross- loss with the
C A DDITIONAL R ELATED W ORK ON F EW -S HOT I MAGE C LASSIFICATION
As one of the unsolved problems in machine learning, few-shot learning (Miller et al., 2000; Fei-Fei et al., 2006) has attracted growing interest in the deep learning era (Lake et al., 2015; Santoro et al., 2016; Wang and Hebert, 2016; Vinyals et al., 2016;
Snell et al., 2017; Finn et al., 2017; Hariharan and
Girshick, 2017; George et al., 2017; Triantaﬁllou et al., 2017; Edwards and Storkey, 2017; Mishra et al., 2018; Douze et al., 2018; Wang et al., 2018; Chen et al., 2019; Dvornik et al., 2019; Allen et al., 2019; Li et al., 2019a; Yoon et al., 2019; Lifchitz et al., 2019; Li et al., 2019b; Zhang et al., 2020; Ye et al., 2020; Dhillon et al., 2020; Triantaﬁllou et al., 2020; Tian et al., 2020; Dvornik et al., 2020; Yang et al., 2021; Phoo and Hariharan, 2021; Liu et al., 2021; Zhang et al., 2021).
Successful generalization from few training samples requires “inductive biases” or shared knowledge from related tasks (Baxter, 1997), which is commonly acquired through transfer learning, and more recently, meta-learning (Thrun, 1998; Schmidhuber, 1987;
Schmidhuber et al., 1997; Bengio et al., 2013).
By explicitly “learning-to-learn” over a series of few-shot learning tasks (i.e., episodes), which are simulated from base classes, meta-learning exploits accumulated task-agnostic knowledge to few-shot learning problems of novel classes.
Within this paradigm, various types of meta-knowledge has been explored, including (1) a generic feature embedding or metric space, in which images are easy to classify using a distance-based classiﬁer such as cosine similarity or nearest neighbor (Koch et al., 2015;
Vinyals et al., 2016;
Snell et al., 2017; Sung et al., 2018; Ren et al., 2018; Oreshkin et al., 2018); (2) a common initialization of network parameters (Finn et al., 2017; Nichol and Schulman, 2018;
Finn et al., 2018) or learned update rules (Andrychowicz et al., 2016; Ravi and Larochelle, 2017; Munkhdalai and Yu, 2017; Li et al., 2017; Rusu et al., 2019); (3) a transferable strategy to estimate model parameters based on few class examples (Bertinetto et al., 2016; Qiao et al., 2018;
Qi et al., 2018; Gidaris and Komodakis, 2018), or from an initial small dataset model (Wang and
Hebert, 2016; Wang et al., 2017).
Some most recent work (Gidaris and Komodakis, 2018; Chen et al., 2019; Tian et al., 2020) also showed that the performance of these complex models can be matched by simple representation learning on base classes and classiﬁer ﬁne-tuning on novel classes.
D D ATASETS AND A DDITIONAL E XPERIMENT D ETAILS
Datasets: mini-ImageNet consists of 64 , 16 , and 20 classes from ImageNet for base, validation, and novel sets, respectively.
Each class contains 600 images of size 84 × 84 .
tiered-ImageNet consists of 351 , 97 , and 160 classes from ImageNet for base, validation, and novel sets, respectively.
Unlike mini-ImageNet, the classes could have varying number of samples in tiered-ImageNet, but the images Number of Shots 0. ( A CC ) % 5-way
ResNet
ResNet ResNet ResNet ResNet
Number of Shots 0.
( A
CC ) % 10-way
ResNet
ResNet
ResNet
ResNet
ResNet are of the same size.
CIFAR-FS is a random split of CIFAR-100 (Krizhevsky et al., 2009) with images of size 32 × 32 into 64 , 16 , and 20 classes for base, validation, and novel sets, respectively.
Setups and Implementation Details: Our experiments fall into two main categories: (1) the standard ResNet (He et al., 2016) feature experiments for logistic classiﬁers, and (2) more powerful, state-of-the-art WideResNet (Zagoruyko and Komodakis, 2016)
features trained with strong regularization techniques (manifold mixup) and additional self-supervision (Mangla et al., 2020), or further calibrated via feature transformations (Yang et al., 2021), for logistic and cosine classiﬁers.
For the ﬁrst category of experiments, we averaged our results over 400 random trials.
An array of 5 different ResNet architectures were used in this category: ResNet10, 18, 34, 50, and 101.
The mini-ImageNet dataset was used for these experiments, and 16-way classiﬁcation was performed on both the validation and novel classes.
For this purpose, 16 out of 20 novel classes were chosen at random once and ﬁxed for all the evaluations.
We split the validation and novel classes into 90% training and 10% held-out for accuracy evaluation.
The imbalanced data settings were chosen to either have 7.5 or 15 average number of shots per class.
For the second category of experiments, we used the WideResNet-28-10 pre-trained feature backbones from Mangla et al.
; Yang et al.
These experiments were conducted on all three benchmarks, and each hyper-parameter conﬁguration was averaged over 10,000 random trials.
Additional details and hyper-parameters are covered in the subsequent sections.
Additional Implementation Details for Sections 4.1 and 4.2 of the Main Paper:
The L2regularization coefﬁcient was chosen for the mean-squared value of all classiﬁer parameters, and the Firth bias reduction coefﬁcient was chosen for Equation (13) of the main paper.
Statistical Signiﬁcance and Reducing the Effect of Randomized Factor: Our study investigates improvements over the baseline.
Random effects (random number seed; batch ordering; parameter initialization; and so on) complicate the study by creating variance in the measured improvement.
We used a matching procedure (so that the baseline and the Firth penalized models share the same values of all random effects) to control this variance.
As long as one does not search for random effects that yield a good improvement (we did not), this yields an unbiased estimate of the improvement.
Each experiment is repeated multiple times to obtain conﬁdence intervals.
Note that (1) conﬁdence intervals are small; and (2) improvements occur over a large range of feature backbones and datasets.
It is safe to conclude that Firth bias reduction reliably offers a small but useful improvement in accuracy for few-shot classiﬁers.
Additional Implementation Details for Section 4.3 of the Main Paper: We used two non-uniform count vectors with different average counts, 7 . 5 (scheme 1) and 15 (scheme 2), to generate the datasets in both validation and novel sets.
The count vector with the average of 7 . 5 -shots had 4 classes for each count from the geometric sequence { 2 , 4 , 8 , 16 } , and the count vector with the average of 15 -shots had 2 classes for each count from the arithmetic sequence { 1 , 5 , 9 , 13 , 17 , 21 , 25 , 29 } .
The same 1-layer logistic classiﬁer of Sections 4.1 and 4.2 was used in Section 4.3 of the main paper.
[Table]
All Standard Backbone Experiments
[Table]
of Sections 4.1 and 4.2 of the
Main Paper
[Table]
of Section 4.3 of the Main
Paper
[Table]
Experiments of
Sections 4.1, 4.2, and 4.3 of the Main Paper with 1-Layer
Logistic Classiﬁer
[Table]
Experiments of
Section E.2 with 3-Layer
Logistic Classiﬁer
[Table]
the normalized set of coefﬁcients seems large.
The typical unnormalized regularization coefﬁcients can be obtained by dividing these normalized coefﬁcients by the number of classiﬁer parameters.
Table A3 summarizes the hyper-parameters used in all the standard backbone experiments.
Also,
Figure A6 shows the relative accuracy improvements corresponding to Figure A5.
Figure A contains the validation accuracy versus Firth coefﬁcient λ for the experiments of Figure 2, and
Figure A5.
Additional Implementation Details for Section 4.5 of the Main Paper:
In the experiments carried out in Section 4.5 of the main paper, a 1-layer cosine classiﬁer was used.
Also, for the Firth bias-reduced cosine classiﬁer, the regularization coefﬁcient was tuned for each ( N, J ) pair, with N representing the number of samples per class and J being the number of classes.
For J -way classiﬁcation on the novel set when J happened to be larger than the number of classes in the validation set ( J val ) the coefﬁcient tuned for J val -way classiﬁcation was adopted.
Computational Resources: For the standard backbone experiments, we trained more than 384, 3-layer and 768,000 1-layer logistic classiﬁers for the balanced data settings.
For the imbalanced settings, we trained more than 64,000 3-layer and 128,000 1-layer logistic classiﬁers.
All classiﬁers were trained on CPUs, and these experiments alone consumed more than 16 CPU years.
Thebackbones were trained once using an Nvidia-V100 GPU and took 3-days for training. [Table] [Table] [Figure]
[Table]
[Figure]
[Table] For the cosine classiﬁer experiments, we trained over 1.92, 1.92, and 3.36 million classiﬁers for mini-ImageNet, CIFAR-FS [Table] and tiered-ImageNet datasets, respectively.
These experiments used more than 5 CPU years in total.
[Table]
[Table]
[Figure]
[Table]
E A DDITIONAL E XPERIMENTS ON THE S TANDARD B ACKBONES E. A DDITIONAL L OGISTIC C LASSIFIER E XPERIMENTS
The experiments of Figure 2 were repeated to perform 16-way classiﬁcation using a logistic classiﬁer on tiered-ImageNet and CIFAR-FS Figure A13.
Moreover, 5-way and 10-way classiﬁcation was tested for mini-ImageNet in Figure A9 in the same setting as Figure 2.
The results show that Firth improvements always exist and it is even more effective as the number of classes increases.
E. 3-L AYER L OGISTIC C LASSIFIERS FOR THE S TANDARD B ACKBONES
We conducted the same experiments as in Sections 4.1 and 4.2 of the main paper but with a 3layer logistic classiﬁer.
Again, we see consistent accuracy improvements for the Firth bias-reduced classiﬁer over the non-penalized (baseline) classiﬁer in Figure A5.
This further supports the idea that Firth bias reduction boosts the performance of any reasonable classiﬁer .
Needless to say,
L2-regularization is not effective as shown in Figure A5.
Furthermore, the imbalanced few-shot classiﬁcation in Section 4.3 of the main paper was repeated with the 3-layer logistic classiﬁer in Figure A10.
Again for both schemes, the Firth penalized classiﬁer has larger accuracy improvement than the classiﬁer penalized with the KL-divergence to the non-uniform prior over the class probabilities.
This further validates the effectiveness of Firth bias reduction in reducing the parameter estimation bias present in the few-shot setting.
F C OMPARING F IRTH B IAS R EDUCTION A GAINST S TANDARD L ABEL S MOOTHING T ECHNIQUES
To demonstrate that Firth bias reduction cannot simply be replaced with label smoothing, we tested two advanced variants of label smoothing that are superior to the original version as proposed by Pereyra et al.
The ﬁrst variant, called conﬁdence penalty, uses the entropy of the classiﬁer’s output (or equivalently, reverses the direction of the KL divergence in the original version of label smoothing (Szegedy et al., 2016)); and the second variant, called unigram label smoothing, uses prior distribution over the classes instead of uniform, which has been shown to be advantageous when the output labels’ distribution is imbalanced in Pereyra et al.
Note that both the variants were investigated for training a full deep neural network with a feature extractor backbone in large-sample [Table] regimes in Pereyra et al.
Surprisingly, our evaluation of these two variants for training only classiﬁers on novel classes in few-shot regimes indicates that they are not as helpful as Firth bias reduction.
Our experiments in Figure 3 show that, in imbalanced few-shot classiﬁcation tasks, the improvements achieved by unigram label smoothing is signiﬁcantly less than the Firth penalized classiﬁer.
This validates the bias suppression effect of Firth – a property that cannot be achieved by unigram label smoothing.
We also performed more experiments in the same setting to compare Firth bias reduction against conﬁdence penalty regularization.
As summarized in Table A4, in all the settings Firth bias reduction has larger signiﬁcant improvements than the conﬁdence penalty technique.
This further supports the value of using Firth bias reduction and the fact that its impact on few-shot classiﬁcation cannot be reproduced with well-known and widely-used label smoothing techniques. [Table]
G A DDITIONAL FEATURE BACKBONES
To test the Firth bias reduction technique for additional backbones, we used pretrained DenseNet and
MobileNet backbones on tiered-ImageNet from (Wang et al., 2019).
The accuracy improvements of Firth penalized logistic classiﬁer over the baseline averaged over 10,000 trials are plotted in
Figure A11.
Regardless of the number of classes, the improvements are always positive
[Table]
H C OMPARISON WITH A DDITIONAL S TATE OF THE A RT METHOD
Table A5 summarizes the accuracy improvements obtained by integrating Firth bias reduction with the distribution calibration method (Yang et al., 2021) under different shots and ways on the tieredImageNet dataset.
This method calibrates the features to follow a normal distribution, and generates
[Figure]
li
[Figure]
i
[Figure]
n
[Figure]
cc u r a c
[Figure]
Logistic
[Figure]
a
[Figure]
d a t i o
[Figure]
A cc u r a c
[Figure] d [Figure] t i o n
A cc u r a c y
[Figure]
artiﬁcial samples from the estimated normal distribution as data augmentation to aid the few-shot classiﬁcation.
In its state-of-the-art setting, the features are transformed using Tukey transformations, 750 artiﬁcial samples are generated per class, and a logistic classiﬁer is used.
We tested Firth bias reduction in two scenarios: (i) state-of-the-art setting without generating artiﬁcial samples from the calibrated distribution; and (ii) state-of-the-art setting with 750 artiﬁcial samples generated per class, as shown in Table A5.
The 95% conﬁdence intervals for the accuracy improvements are reported in both cases. Number of Shots
[Figure] Firth Bias Reduction (tiered-ImageNet) ResNet ResNet
ResNet
ResNet
ResNet
Number of Shots
[Figure]
L2 Regularization (tiered-ImageNet)
ResNet
ResNet
ResNet
ResNet
ResNet
Number of Shots
[Figure]
[Figure]
Reduction (CIFAR-FS)
ResNet
ResNet
ResNet
ResNet
ResNet
Number of Shots
[Figure] L2 Regularization (CIFAR-FS) ResNet ResNet
ResNet
ResNet ResNet
16-way logistic classiﬁcation was conducted in this experiment, with over 1,000 randomized and matching trials for each combination of method, backbone and number of samples.
As shown in Table A5, Firth bias reduction produces positive improvements in all cases, which are statistically signiﬁcant in all the cases.
As expected, Firth bias reduction produces larger improvements when artiﬁcial sample generation is disabled, and thus there is a larger maximum likelihood estimation bias on the linear classiﬁer (despite the use of Tukey transformation).
In the case that artiﬁcial samples are added and they are tuned to follow the original normal distribution, havingof them can to some extent alleviate the bias in the estimation of the logistic classiﬁer’s weights in the few-shot regime.
However, the results show that even in the presence of more data (artiﬁcial samples), Firth bias reduction is still effective.
This suggests that producing artiﬁcial samples to augment data cannot resolve the estimation bias issue, as they are more likely to be similar to the limited real samples available.
We suspect this will become an even more severe problem in the cross-domain few-shot setting; producing artiﬁcial samples will be signiﬁcantly less effective than
Firth bias reduction, due to the domain shift (see Section 4.6).