TransBoost: Improving the Best ImageNet Performance using Deep Transduction
Abstract
This paper deals with deep transductive learning , and proposes TransBoost as a procedure for fine-tuning any deep neural model to improve its performance on any (unlabeled) test set provided at training time.
TransBoost is inspired by a large margin principle and is efficient and simple to use.
Our method significantly improves the ImageNet classification performance on a wide range of architectures, such as ResNets, MobileNetV3-L, EfficientNetB0, ViT-S, and ConvNext-T, leading to state-of-the-art transductive performance.
Additionally we show that TransBoost is effective on a wide variety of image classification datasets.
The implementation of TransBoost is provided at: https://github.com/omerb01/TransBoost .
Introduction
Alternative learning frameworks that make use of unlabeled data have gained considerable attention in recent years.
These include semi-supervised learning and transductive learning , both of which were extensively studied in classical and statistical machine learning [1, 2, 3] before the advent of deep learning.
In this article we focus on deep transductive classification.
In this setting, both a labeled training sample and an (unlabeled) test sample are provided at training time.
The goal is to predict only the labels of the given test instances as accurately as possible.
In contrast, in standard * inductive classification, the goal is to train a general model capable of predicting the labels of unseen test instances.
In semi-supervised learning (SSL) we are also given a set of unlabeled examples, which is used for selecting a model.
Despite a commonly held misconception, however, the goal of SSL remains inductive rather than transductive, and the trained model must predict labels for new instances (i.e., the unlabeled instances are used for training only).
Transductive models deliver higher accuracy gains compared with traditional inductive models, as we demonstrate in this paper in the context of image classification.
Much of the power of transductive learning is achieved through analyzing the given test set as a group (see Section 5.4).
Transductive prediction is thus especially useful whenever we can accumulate a test set of instances and then train a specialized model to predict their labels.
While there are many possible relevant use cases, one that stands out is medical diagnosis.
Here, daily or weekly medical records can be accumulated and sent to the transductive predictor as a set.
In Appendix A, we highlight a number of additional use cases.
Perhaps the best-known transductive classification method in classical machine learning is the transductive support vector machine (TSVM)
While a support vector machine (SVM) [6] seeks to achieve a general decision function, a TSVM attempts to reduce misclassifications of just the target test instances.
To the best of our knowledge, in deep learning, transductive settings have only been addressed in the case of few-shot learning (t-FSL).
In this paper we address deep transductive learning and introduce TransBoost , a novel procedure that takes a pretrained neural classification model, along with a (labeled) training set, and an (unlabeled) test set.
The procedure fine-tunes the model to improve its performance on this particular test set.
The
TransBoost optimization procedure is inspired by TSVM and its objective attempts to approximate a large margin principle .
TransBoost is implemented through a simple transductive loss component, which is combined with the standard cross-entropy loss.
We examine the effectiveness of TransBoost by using a variety of pretrained neural networks, datasets, and baseline models.
Our extensive study examines a spectrum of ResNet architectures and a variety of modern architectures such as ConvNext [7], MobileNetv3 [8] and ViT [9].
Figure 1 highlights some of TransBoost’s results on the ImageNet dataset, which demonstrate impressive and consistent accuracy gains, leading to state-of-the-art performance compared to standard (inductive) classification as well as versus SSL and t-FSL methods (see full results in Section 5.2).
To summarize, the contributions of this paper are: (1) A novel deep transductive loss function, inspired by a large margin principle, which optimizes performance on a specific test set.
A simple and efficient transductive procedure for fine-tuning a pretrained neural model in order to boost its performance over an inductive setting.
(3) A comprehensive empirical study of the proposed fine-tuning procedure showing great benefits across a large number of architectures, datasets and baseline techniques.
Problem Formulation
As transductive learning has not been extensively researched in the deep learning community, we begin by defining the transductive learning problem within the context of deep learning classification.
We follow Vapnik’s standard formulation from statistical learning theory [4].
Let P ( X, Y ) be a probability distribution over X × Y , where X represents an input space (e.g., raw image data), and
Y represents a label set corresponding to C classes.
The learner is provided with a set of labeled instances, S l ≜ { ( x 1 , y 1 ) , . . .
, ( x L , y L ) } , where x i ∈
X , y i ∈ Y , and a finite set of unlabeled instances, X u ≜ { x L +1 , . . .
, x L + U } , where x i ∈ X .
The objective is to label the unlabeled instances based on this data.
Given a deep neural network f θ : X → Y , where θ denotes its parameters, and a loss function ℓ : Y × Y → R , we follow [1] (Setting 2) and define the (true) risk over the unlabeled set,
[Equation]
Thus, we are given a labeled training set S l ≜ ( X l , Y l ) selected i.i.d.
according to P ( X, Y ) , as well as a pretrained classification model f θ , and we presume it was trained with the labeled set S l .
An independent test set S u ≜ ( X u , Y u ) , of U samples is randomly and independently selected in the same manner, and we are required to revise the given model based on both S l and X u , so as to minimize Equation (1) without knowing the labels Y u .
Related Work
Transductive learning (or transduction ) was first formulated by Vapnik [4, 10] who also introduced the first transductive algorithm – TSVM [4, 5], which learns a large margin hyperplane classifier from labeled training data while simultaneously forcing it to take into account the (unlabeled) test data.
Whereas transduction received considerable attention in the context of classical machine learning [1, 2, 3], it has only been briefly addressed in the context of deep neural networks.
Recall that in (deep) transductive classification we are provided with both a labeled training sample and an unlabeled test sample.
A related but different setting is semi-supervised learning (SSL), where in addition to the labeled training set, we are also given a (typically large) unlabeled training sample.
The objective is inductive – namely, to guess the label of any unseen test sample (that was drawn from the same distribution).
A simple observation is that any SSL algorithm can be applied in transduction by using the given (unlabeled) test sample as the unlabeled SSL training sample.
Thus, algorithms such as [11, 12, 13] can be used to solve transductive learning.
This fact may explain a common confusion between SSL and transduction.
Nevertheless, an inductive SSL algorithm does not need to use (and cannot rely on) the fact that it will only be asked about the given test points.
In contrast, a (meaningful) transductive algorithm must rely on this knowledge, as we contend in this paper.
In fact, in Section 5.2 we demonstrate that our proposed transductive procedure substantially outperforms the
SimCLR [11] and SimCLRv2 [12] algorithms, where SimCLRv2 currently achieves state-of-the-art
SSL performance according to [14].
Recently, there has been a flurry of research concerning transductive few-shot learning (t-FSL) [15, 16, 17].
In t-FSL we are given a classifier for a certain classification task, such as cats versus dogs, as well as a training set (called the support set) for unseen classes (referred to as ways ), such as elephants, tigers and lizards, containing only few labeled samples per unseen class.
A test set (called the query set), containing instances from the unseen classes, is given and the goal is to learn the additional classes so as to correctly classify these test instances.
Clearly, this setting is transductive; however, the goal here is to learn from a few labeled examples (e.g., one-shot, five-shot).
Formally, a t-FSL algorithm can be applied to transductive classification (by treating the training set as a support set).
For the most part, however, methods for t-FSL cannot be effectively applied to large (non few-shot) training sets (e.g., iLPC
Nevertheless, in order to conduct a complete study in transductive classification, we chose two t-FSL representative algorithms as baselines for our experiments (see Section 5).
Specifically, we consider the entropy minimization (Ent-min) method of [15], and the TIM-GD method of [16].
With Ent-min, the Shannon entropy is reduced by fine-tuning, whereas TIM-GD optimizes the mutual information of test instances using only the classifier weights with fine-tuning.
Ent-min is a common t-FSL baseline while TIM-GD is a top performing 5-shot t-FSL algorithm according to the few-shot leaderboard [18].
TransBoost: Fine-Tuning via Transductive Learning
We now present TransBoost , a procedure that takes a pretrained neural model along with its (labeled) training set as well as an (unlabeled) test set.
The procedure then fine-tunes the model to improve its performance for this specific test set.
The TransBoost optimization objective, L , is driven by
Equation (3) and approximated by Algorithm 1, which includes a novel transductive loss component in addition to the standard cross-entropy loss.
Our transductive component, L TransBoost , is introduced in Equation (3) as a regularization term and is defined in Equation (2).
L TransBoost is inspired by
TSVMs [4, 5] and follows a large margin principle .
Let us elaborate on our loss function, L TransBoost (2), which is applied on the set of test samples, X u .
This loss function includes an unsupervised non-negative symmetric pairwise similarity function,
S :
X × X
→ R , which measures the similarity of two input instances.
In our implementation, we simply used an L 2 -based score function applied on prediction (softmax) vectors generated by f θ ; see Equation (4).
S is only applied on pairs of test instances that are likely to belong to different classes.
The likelihood is determined by using an implementation of the Kronecker delta function, δ :
X × X → { 0 , 1 } , which obtains 1 iff the instances are predicted to belong to different classes; see Equation (5) for our implementation.
Additionally, for each pair we intensify the loss using the model’s confidence in its predictions.
We define κ : X → R + to be a confidence function, which for each instance, x , gives its class prediction confidence; see Equation (6) for our implementation.
Finally, our transductive loss function, L TransBoost , is
[Equation]
where U δ ≜ (cid:80) 1 ≤ i<j ≤ U δ ( x i , x j ) .
The final loss function of our fine-tuning procedure includes the transductive loss term (2) as a regularization term as follows,
[Equation]
where ℓ is a standard (inductive) pointwise loss, e.g., the cross-entropy loss and λ ∈ R is a regularization hyperparameter.
Using the optimization objective in Equation (3), we incentivize test sample pairs that are likely to be different in their classes to also be different in their empirical class probabilities while preserving the prior knowledge of f θ .
Accordingly, given two test samples, ( x i , x j ) , which are predicted to be different in their classes by δ , we obtain a higher cost (by S ) if x i is close (similar) to x j in its prediction vector than if x i is far (dissimilar) from x j , depending on the model’s confidence in its predictions utilizing κ .
The computation of the L TransBoost loss component (2) is quadratic in U – namely, its time complexity is O ( (cid:0) U 2 (cid:1) ) .
For large test sets (e.g., ImageNet), this is prohibitively large.
Therefore, we propose to approximate L TransBoost by using random subsets of pairs from the test set.
This random sampling is applied for each minibatch.
The pseudo-code of the optimization procedure is presented in
Algorithm 1.
As can be seen, the size of the set of sampled test pairs used to approximate (2) is taken to be the batch size ( U ′ ), which allows for very fast computation of L TransBoost but could be sub-optimal (this size was not optimized).
4.1 TransBoost Implementation
While S , δ and κ in Equation (2) can be implemented in many ways, we now describe our proposed implementation that was used to obtain all the results described in Section 5.
As can be seen below, we attempted to instantiate Equation (2) in the simplest possible manner.
The consideration of more sophisticated methods is left for future work (and an alternative choice is discussed in Section 5.5).
The unsupervised symmetric pairwise similarity function S f was straightforwardly implemented using the L 2 norm as follows,
[Equation]
where ˆ p ( x | f θ ) denotes the empirical class probability of f θ applied on x .
It is easy to show that S f is non-negative * (see Appendix B for a proof).
Additionally, we implemented δ f (5) and κ f (6) based on the pretrained weights of the model before it was fine-tuned with our procedure, which we refer to as θ 0 .
Thereafter, the unsupervised symmetric pairwise selection function δ f is implemented using pseudo-labels computed by the given pretrained model f θ 0 ,
[Equation]
and the confidence function κ f is implemented using the standard softmax response, i.e.,
[Equation]
where ˆ p ( x | f θ 0 ) is the empirical class probabilities of f θ 0 applied to x , and C is the number of classes.
Finally, the optimization objective (3) is implemented using the standard cross-entropy loss,
CE ( X l , Y l | f θ ) ≜
1 L (cid:80) Li
=1 − log ˆ p y
i
( x i | f θ ) , as well the proposed S f , δ f and κ f ,
[Equation]
where λ ∈ R is a regularization hyperparameter.
4.2 A Large Margin Analogy of TransBoost
TransBoost is inspired by TSVM, in which a large margin principle is applied to the given (unlabeled) test sample to leverage the knowledge of this specific test set.
In Figure 2 we present a toy quantitative (graphical) example indicating that our transductive loss component, L TransBoost , encourages behavior similar to TSVM.
Consider Figure 2 (left) showing the decision boundary obtained by SVM for the given training set (and ignoring the test points).
Using SVM to implement δ (to determine if two test points are in the same class), we get that L TransBoost = 0 .
1796 (shown at the bottom).
Clearly, this loss value is significantly larger than the corresponding loss (0.05167) obtained when using TSVM (right), where a large margin principle of test instances is applied.
Empirical Study
In this section we present a comprehensive empirical study of TransBoost.
We begin by describing our experimental design and then present our studies in detail.
5.1 Experimental Design and Details
This section describes the experimental details: datasets used, architectures used, and our TransBoost’s procedure details and hyperparameters.
Datasets and Preprocessing.
Most of our study of TransBoost is done using the well-known
ImageNet-1k ILSVRC-2012 dataset [19], which contains 1,281,167 training instances and 50, test instances in 1,000 categories.
Preprocessing for training is standard and includes resizing and random cropping to 224 × 224, followed by a random horizontal flip.
For testing, the images are resized and center cropped.
In some of our experiments involving advanced architectures from the
Timm repository [20], we applied the test augmentations (at test time only) using its code from the
Timm repository.
Additionally, we investigated the effectiveness of TransBoost on the Food- [21], CIFAR-10 and CIFAR-100 [22], SUN-397 [23], Stanford Cars [24], FGVC Aircraft [25], the
Describable Textures Dataset (DTD) [26] and Oxford 102 Flowers [27].
For each of these datasets, we applied the same preprocessing as in ImageNet, with the exception of CIFAR-10 and CIFAR-100, where we used random cropping with padding (cropping to 32 × 32), followed by a random horizontal flip, a single RandAugment [28] operation and MixUp [29].
Architectures.
We examined various inductive (standard) pretrained architectures as baselines for TransBoost.
Specifically, we used a variety of ResNet [30] architectures: ResNet18, ResNet34,
ResNet50, ResNet101 and ResNet152, which were pretrained using the standard PyTorch training recipe [31].
We also considered ResNet50-StrikesBack [32], which is the ResNet50 architecture trained with Timm’s [20] advanced training recipe.
Additionally, we show results with more advanced architectures spanning various families: EfficientNetB0 [33] and MobileNetV3-L [8], ConvNext-T [7], and the Transformer models ViT-S [9] and Swin-T [34], which were pretrained with advanced procedures of Timm’s library [20] in accordance to their original papers.
Details on the TransBoost Fine-Tuning Procedure.
Throughout all TransBoost’s experiments, we followed Algorithm 1, and our functions were implemented as described in Section 4.1.
We used the SGD optimizer (Nesterov) with a momentum of 0.9, weight decay of 10 − 4 , and a batch size of 1024 (consisting of 512 labeled training instances and 512 unlabeled test instances).
Unless otherwise specified, the learning rate was fixed to 10 − 3 with no warmup for 120 epochs.
The regularization hyperparameter of our loss in all our experiments was fixed to λ = 2 ; see Equation (7).
Our hyperparameters were fine-tuned based on a validation set that was sampled from the training set over ImageNet.
5.2 ImageNet Experiments: Transductive vs. Inductive
This study examines the transductive performance of TransBoost applied to various pretrained models (all described in Section 5.1), and the transductive performance of the other baselines (SSL and t-FSL) versus the inductive performance of all baselines.
Table 1 presents all the results.
We note that some of these results have been already highlighted in Figure 1.
Table 1 is divided into several horizontal sections.
In the first section we present TransBoost applied to a number of ResNet architectures.
In the second section, we discuss modern architectures, including two types of Vision Transformers, and also ConvNext-T as representative of the ConvNext family, which presently achieves state-of-the-art results over ImageNet [14].
Next we present the
SimCLR and SimCLRv2 SSL methods (see Appendix C for more details on how we trained them in transduction).
We note that SimCLRv2 presently dominates SSL performance over ImageNet according to [14].
The last table section considers two popular t-FSL methods (which are described in Section 3).
The column structure of Table 1 is straightforward but the most important column that deserves some explanation is the Inductive / Transductive column ( 4 th column) in which we compare the transductive performance of each experiment to the relevant inductive fully supervised performance.
For example, TransBoost is capable of increasing the ImageNet top-1 accuracy of
ViT-S by 2.28%.
Table 1 and Figure 1 demonstrate that fine-tuning with TransBoost consistently and significantly improves the inductive top-1 accuracy performance in all cases.
Even the tiny version of the top performing ConvNext architecture family (presently the state-of-the-art according to [14]) is improved by TransBoost.
Surprisingly, ViT-S performed better than ConvNext-T when taking advantage of the given test samples using TransBoost, indicating that members of the ViT family could be the best transductive performers on ImageNet.
A second interesting fact is that ViT-S and Swin-T
[Table] [Table]
[Table]
perform similarly in the inductive setting while the ViT-S top-1 accuracy gain is improved by almost x3 relative to Swin-T. Could this advantage be related to the architecture?
Interestingly,
TransBoost also improves the performance of ResNet50-StrikesBack, which is the standard ResNet architecture trained using a sophisticated training procedure [32] that boosts its top-1 accuracy by +4.2%.
Surprisingly, TransBoost adds almost +1% on top of this phenomenal improvement.
Another striking related result is that TransBoost improves the standard ResNet50 so that it achieves performance close to ResNet50-StrikesBack despite the fact that our procedure is much simpler than the sophisticated training procedure of [32].
Consider the first section of Table 1 (the ResNets section).
While TransBoost always improves the inductive baselines, it is evident that its relative advantage monotonically decreases with the model size/performance.
For example, the best transductive improvement is achieved for ResNet yielding a top-1 accuracy gain of +3.6% and resulting in a top-1 accuracy of 73.36% * .
With respect to the other related baselines, TransBoost consistently outperforms both the SSL and t-FSL baseline algorithms.
Comparing the results of SimCLR to the results of SimCLRv2, we see that SimCLR outperforms SimCLRv2 in the transductive setting (they mainly differ by the additional stage of knowledge distillation [36] that is added to SimCLRv2), which suggests that the stage of knowledge distillation can hinder the transductive performance.
We hypothesize that distilling knowledge from test instances leads to overfitting of incorrect class predictions that lead to model confusion.
Finally, when evaluating the t-FSL representatives, we observe that Ent-min improves its inductive baseline while TIM-GD (which is an upgraded version of Ent-min) is inferior to its inductive baseline.
5.3 TransBoost’s Performance on Additional Vision Datasets
We now examine the performance of TransBoost across eight other datasets (all described in Section 5.1).
Throughout this study, we start with a ResNet50 architecture pretrained on ImageNet (using
PyTorch’s standard training recipe).
Then we apply inductive fine-tuning for each specific dataset and, finally, we apply our TransBoost procedure (see Appendix D.1 for more details).
The results are visually depicted in the graph of Figure 3 and are also shown in Table S.1 in the
Appendix.
Clearly, TransBoost outperforms its inductive baseline in all cases.
The improvement is large and significant in all datasets with the exception of CIFAR-10 where the improvement is minor.
The most prominent result is obtained for the FGVC Aircraft dataset, where we observe a striking +5.34% top-1 accuracy gain from 78.46% to 83.8%.
For a more in-depth discussion, we refer the reader to Appendix E.
5.4 Transductive Performance Improves with Test Set Size
In this study, we consider various transductive settings, where each setting is characterized by a certain training set size and a certain test set size.
We are interested in examining the final ImageNet transductive behavior of ResNet50 as a function of these sizes.
The sizes of our training sets are: 5%, 10%, 20%, 100%; and for testing sets are: 10%, 25%, 50%, 100%, all taken as a fraction of the original train/validation set sizes, respectively.
Consider Figure 4 showing the top-1 accuracy gains for all combinations ( 4 × 4 experiments in total).
Importantly, we observe that whenever TransBoost uses the entire training set, its performance increases as the test set size grows.
This strongly indicates that TransBoost leverages the test set as a whole, a desirable property of a transductive learning algorithm.
On the other hand, and quite surprisingly, when TransBoost only used 5% of the training set, increasing the test set size worsened its performance.
We hypothesize that this type of behavior is a result of poor prior knowledge (very small training set) that leads the models to wrongly analyze the relationships (similarities in our case) between pairs.
This overall bad effect increases as the test set size expands.
For a detailed description/discussion of this section’s experiments, see Appendix E.
5.5 Bringing the Like Together or Separating the Difference?
TransBoost is driven by Equation (2) whose informal desideratum is: revise the model so as to take apart the representations of two instances that are likely ( κ ) to belong to different classes ( δ ), which currently appear to be similar ( S ).
Rather than, or in addition to separating instances that are likely to be different, we could consider grouping together instances that are likely to belong to the same class.
In this section we summarize our experiments to apply TransBoost for this alternative objective and for a combination of both objectives.
In our study we used the ResNet50 architecture, which was pretrained on ImageNet (using PyTorch’s standard training recipe).
The transductive performance of TransBoost was analyzed using three variants of the transductive loss component: (1) the proposed separation component, L TransBoost ( X u |S f , δ f , κ f ) , as in Equation (7); (2) a “reciprocal” instantiation of this objective – namely, L TransBoost ( X u |√ 2 − S f , 1 − δ f , κ f ) , which brings together similar instances; and (3) a variant that considers both these objectives together.
The results are presented in Table 2.
Transductive
[Table]
Clearly (but not significantly) the best performance is achieved using our proposed implementation (row highlighted in green).
We chose our proposed loss component over the combination of both objectives because it is simpler.
Interestingly, the reciprocal variation significantly degrades the topperformance.
Concluding Remarks
We presented TransBoost, a novel and powerful transductive fine-tuning procedure that can be efficiently applied on any pretrained model.
TransBoost is a deep transductive classification algorithm that is inspired by a large margin principle such as the TSVM algorithm.
Strikingly, TransBoost consistently and significantly improves the inductive ImageNet top-1 performance of many architectures including the most advanced ones, leading to state-of-the-art transductive performance.
Moreover,
TransBoost is effective on a broad range of image classification datasets.
As described in Algorithm 1, TransBoost is a general optimization procedure that can be implemented in a range of ways depending on the similarity function ( S ), the selection function ( δ ), and the confidence function ( κ ).
We instantiated these functions simply and straightforwardly.
A question for future research may be to ask whether more sophisticated choices will lead to better transduction performance.
The ViT-S architecture achieved the best transductive classification performance in our experiments and accrued the largest gain relative to induction among advanced architectures.
It would be interesting to explore the question of how this Transformer architecture facilitates this gain.
In general, we can ask what is the best architecture for transductive classification.
Finally, it would be very interesting to see whether such large performance gains are also sustained in NLP applications.
Finally, our results clearly demonstrate that transductive learning can lead to huge accuracy gains.
Thus, it would be wasteful to use standard induction in settings where transduction is applicable.
Acknowledgments
This research was partially supported by the Israel Science Foundation, grant No. 710/18.