TransBoost: Improving the Best ImageNet Performance using Deep Transduction
Anonymous Author(s)
AffiliationAddress email
Abstract
This paper deals with deep transductive learning , and proposes TransBoost as aprocedure for fine-tuning any deep neural model to improve its performance onany (unlabeled) test set provided at training time.
TransBoost is inspired by alarge margin principle and is efficient and simple to use.
The ImageNet classification performance is consistently and significantly improved with TransBoost onmany architectures such as ResNets, MobileNetV3-L, EfficientNetB0, ViT-S and
ConvNext-T.
Additionally we show that TransBoost is effective on a wide varietyof image classification datasets.
Introduction
Alternative learning frameworks that make use of unlabeled data have gained considerable attentionin recent years.
These include semi-supervised learning and transductive learning , both of whichwere extensively studied in classical and statistical machine learning [1, 2, 3] before the advent ofdeep learning.
In this article we focus on deep transductive classification.
In this setting, both alabeled training sample and an (unlabeled) test sample are provided at training time.
The goal is topredict only the labels of the given test instances as accurately as possible.
In contrast, in standardinductive classification, the goal is to train a general model capable of predicting the labels of unseentest instances.
In semi-supervised learning (SSL) we are also given a set of unlabeled examples, which is usedfor selecting a model.
Despite a commonly held misconception, however, the goal of SSL remainsinductive rather than transductive, and the trained model must predict labels for new instances (i.e.,the unlabeled instances are used for training only).
Transductive models deliver higher accuracy gains compared with traditional inductive models, as wedemonstrate in this paper in the context of image classification.
Much of the power of transductivelearning is achieved through analyzing the given test set as a group (see Section 5.4).
Transductiveprediction is thus especially useful whenever we can accumulate a test set of instances and then traina specialized model to predict their labels.
While there are many possible relevant use cases, one thatstands out is medical diagnosis.
Here, daily or weekly medical records can be accumulated and sentto the transductive predictor as a set.
Perhaps the best-known transductive classification method in classical machine learning is thetransductive support vector machine (TSVM)
While a support vector machine (SVM) [6]seeks to achieve a general decision function, a TSVM attempts to reduce misclassifications of just thetarget test instances.
To the best of our knowledge, in deep learning, transductive settings have onlybeen addressed in the case of few-shot learning (t-FSL).
In this paper we address deep transductive learning and introduce TransBoost , a novel procedure thattakes a pretrained neural classification model, along with a (labeled) training set, and an (unlabeled)test set.
The procedure fine-tunes the model to improve its performance on this particular test set.
The
TransBoost optimization procedure is inspired by TSVM and its objective attempts to approximate alarge margin principle .
TransBoost is implemented through a simple transductive loss component,which is combined with the standard cross-entropy loss.
We examine the effectiveness of TransBoost by using a variety of pretrained neural networks, datasets,and baseline models.
Our extensive study examines a spectrum of ResNet architectures and a varietyof modern architectures such as ConvNext [7], MobileNetv3 [8] and ViT [9].
Figure 1 highlightssome of TransBoost’s results on the ImageNet dataset, which demonstrate impressive and consistentaccuracy gains, leading to state-of-the-art performance compared to standard (inductive) classificationas well as versus SSL and t-FSL methods (see full results in Section 5.2).
To summarize, the contributions of this paper are: (1) A novel deep transductive loss function,inspired by a large margin principle, which optimizes performance on a specific test set.
(2) Asimple and efficient transductive procedure for fine-tuning a pretrained neural model in order to boostits performance over an inductive setting.
(3) A comprehensive empirical study of the proposedfine-tuning procedure showing great benefits across a large number of architectures, datasets andbaseline techniques.
Problem Formulation
As transductive learning has not been extensively researched in the deep learning community, webegin by defining the transductive learning problem within the context of deep learning classification.
We follow Vapnik’s standard formulation from statistical learning theory [4].
Let P ( X, Y ) be aprobability distribution over X × Y , where X represents an input space (e.g., raw image data), and
Y represents a label set corresponding to C classes.
The learner is provided with a set of labeledinstances, S l ≜ { ( x 1 , y 1 ) , . . .
, ( x L , y L ) } , where x i ∈
X , y i ∈ Y , and a finite set of unlabeledinstances, X u ≜ { x L +1 , . .
, x L + U } , where x i ∈ X .
The objective is to label the unlabeled instancesbased on this data.
Given a deep neural network f θ : X → Y , where θ denotes its parameters, and aloss function ℓ : Y × Y → R , we follow [1] (Setting 2) and define the (true) risk over the unlabeledset,
[Equation]
Thus, we are given a labeled training set S l ≜ ( X l , Y l ) selected i.i.d.
according to P ( X, Y ) , aswell as a pretrained classification model f θ , and we presume it was trained with the labeled set S l .
An independent test set S u ≜ ( X u , Y u ) , of U samples is randomly and independently selected inthe same manner, and we are required to revise the given model based on both S l and X u , so as tominimize Equation (1) without knowing the labels Y u .
Related Work
Transductive learning (or transduction ) was first formulated by Vapnik [4, 10] who also introducedthe first transductive algorithm – TSVM [4, 5], which learns a large margin hyperplane classifier fromlabeled training data while simultaneously forcing it to take into account the (unlabeled) test data.
Whereas transduction received considerable attention in the context of classical machine learning[1, 2, 3], it has only been briefly addressed in the context of deep neural networks.
Recall that in (deep) transductive classification we are provided with both a labeled training sampleand an unlabeled test sample.
A related but different setting is semi-supervised learning (SSL), wherein addition to the labeled training set, we are also given a (typically large) unlabeled training sample.
The objective is inductive – namely, to guess the label of any unseen test sample (that was drawn fromthe same distribution).
A simple observation is that any SSL algorithm can be applied in transductionby using the given (unlabeled) test sample as the unlabeled SSL training sample.
Thus, algorithmssuch as [11, 12, 13] can be used to solve transductive learning.
This fact may explain a commonconfusion between SSL and transduction.
Nevertheless, an inductive SSL algorithm does not need touse (and cannot rely on) the fact that it will only be asked about the given test points.
In contrast, a(meaningful) transductive algorithm must rely on this knowledge, as we contend in this paper.
In fact,in Section 5.2 we demonstrate that our proposed transductive procedure substantially outperforms the
SimCLR [11] and SimCLRv2 [12] algorithms, where SimCLRv2 currently achieves state-of-the-art
SSL performance according to [14].
Recently, there has been a flurry of research concerning transductive few-shot learning (t-FSL)[15, 16, 17].
In t-FSL we are given a classifier for a certain classification task, such as cats versusdogs, as well as a training set (called the support set) for unseen classes (referred to as ways ), such aselephants, tigers and lizards, containing only few labeled samples per unseen class.
A test set (calledthe query set), containing instances from the unseen classes, is given and the goal is to learn theadditional classes so as to correctly classify these test instances.
Clearly, this setting is transductive;however, the goal here is to learn from a few labeled examples (e.g., one-shot, five-shot).
Formally,a t-FSL algorithm can be applied to transductive classification (by treating the training set as asupport set).
For the most part, however, methods for t-FSL cannot be effectively applied to large(non few-shot) training sets (e.g., iLPC
Nevertheless, in order to conduct a complete studyin transductive classification, we chose two t-FSL representative algorithms as baselines for ourexperiments (see Section 5).
Specifically, we consider the entropy minimization (Ent-min) method of[15], and the TIM-GD method of [16].
With Ent-min, the Shannon entropy is reduced by fine-tuning,whereas TIM-GD optimizes the mutual information of test instances using only the classifier weightswith fine-tuning.
Ent-min is a common t-FSL baseline while TIM-GD is a top performing 5-shott-FSL algorithm according to the few-shot leaderboard [18].
TransBoost: Fine-Tuning via Transductive Learning
We now present TransBoost , a procedure that takes a pretrained neural model along with its (labeled)training set as well as an (unlabeled) test set.
The procedure then fine-tunes the model to improveits performance for this specific test set.
The TransBoost optimization objective, L , is driven by
Equation (3) and approximated by Algorithm 1, which includes a novel transductive loss componentin addition to the standard cross-entropy loss.
Our transductive component, L TransBoost , is introducedin Equation (3) as a regularization term and is defined in Equation (2).
L TransBoost is inspired by
TSVMs [4, 5] and follows a large margin principle .
Let us elaborate on our loss function, L TransBoost (2), which is applied on the set of test samples, X u .
This loss function includes an unsupervised non-negative symmetric pairwise similarity function,
S :
X × X
→ R , which measures the similarity of two input instances.
In our implementation, wesimply used an L 2 -based score function applied on prediction (softmax) vectors generated by f θ ;see Equation (4).
S is only applied on pairs of test instances that are likely to belong to differentclasses.
The likelihood is determined by using an implementation of the Kronecker delta function, δ :
X × X → { 0 , 1 } , which obtains 1 iff the instances are predicted to belong to different classes;see Equation (5) for our implementation.
Additionally, for each pair we intensify the loss using themodel’s confidence in its predictions.
We define κ : X → R + to be a confidence function, whichfor each instance, x , gives its class prediction confidence; see Equation (6) for our implementation.
Finally, our transductive loss function, L TransBoost , is
[Equation]
where U δ ≜ (cid:80) 1 ≤ i<j ≤ U δ ( x i , x j ) .
The final loss function of our fine-tuning procedure includes thetransductive loss term (2) as a regularization term as follows,
[Equation]
where ℓ is a standard (inductive) pointwise loss, e.g., the cross-entropy loss and λ ∈ R is a regularization hyperparameter.
Using the optimization objective in Equation (3), we incentivize test sample pairs that are likely to bedifferent in their classes to also be different in their empirical class probabilities while preservingthe prior knowledge of f θ .
Accordingly, given two test samples, ( x i , x j ) , which are predicted tobe different in their classes by δ , we obtain a higher cost (by S ) if x i is close (similar) to x j in itsprediction vector than if x i is far (dissimilar) from x j , depending on the model’s confidence in itspredictions utilizing κ .
The computation of the L TransBoost loss component (2) is quadratic in U – namely, its time complexityis O ( (cid:0) U 2 (cid:1) ) .
For large test sets (e.g., ImageNet), this is prohibitively large.
Therefore, we propose toapproximate L TransBoost by using random subsets of pairs from the test set.
This random samplingis applied for each minibatch.
The pseudo-code of the optimization procedure is presented in
Algorithm 1.
As can be seen, the size of the set of sampled test pairs used to approximate (2) istaken to be the batch size ( U ′ ), which allows for very fast computation of L TransBoost but could besub-optimal (this size was not optimized).
4.1 TransBoost Implementation
While S , δ and κ in Equation (2) can be implemented in many ways, we now describe our proposedimplementation that was used to obtain all the results described in Section 5.
As can be seen below,we attempted to instantiate Equation (2) in the simplest possible manner.
The consideration of moresophisticated methods is left for future work (and an alternative choice is discussed in Section 5.5).
The unsupervised symmetric pairwise similarity function S f was straightforwardly implementedusing the L 2 norm as follows,
[Equation]
where ˆ p ( x | f θ ) denotes the empirical class probability of f θ applied on x .
It is easy to show that S f isnon-negative 1 (see Appendix A for a proof).
Additionally, we implemented δ f (5) and κ f (6) based on the pretrained weights of the model beforeit was fine-tuned with our procedure, which we refer to as θ 0 .
Thereafter, the unsupervised symmetricpairwise selection function δ f is implemented using pseudo-labels computed by the given pretrainedmodel f θ 0 ,
[Equation]
and the confidence function κ f is implemented using the standard softmax response, i.e.,
[Equation]
6) where ˆ p ( x | f θ 0 ) is the empirical class probabilities of f θ 0 applied to x , and C is the number of classes.
Finally, the optimization objective (3) is implemented using the standard cross-entropy loss,
CE ( X l , Y l | f θ ) ≜
1 L (cid:80) Li
=1 − log ˆ p y
i
( x i | f θ ) , as well the proposed S f , δ f and κ f ,
L ( X l , Y l , X u
[Equation]
where λ ∈ R is a regularization hyperparameter.
4.2 A Large Margin Analogy of TransBoost
[Figure]
TransBoost is inspired by TSVM, in which a large margin principle is applied to the given (unlabeled)test sample to leverage the knowledge of this specific test set.
In Figure 2 we present a toy quantitative(graphical) example indicating that our transductive loss component, L TransBoost , encourages behaviorsimilar to TSVM.
Consider Figure 2 (left) showing the decision boundary obtained by SVM for thegiven training set (and ignoring the test points).
Using SVM to implement δ (to determine if two testpoints are in the same class), we get that L TransBoost = 0 .
1796 (shown at the bottom).
Clearly, thisloss value is significantly larger than the corresponding loss (0.05167) obtained when using TSVM(right), where a large margin principle of test instances is applied.
Empirical Study
In this section we present a comprehensive empirical study of TransBoost.
We begin by describingour experimental design and then present our studies in detail.
5.1 Experimental Design and Details
This section describes the experimental details: datasets used, architectures used, and our TransBoost’sprocedure details and hyperparameters.
Datasets and Preprocessing.
Most of our study of TransBoost is done using the well-known
ImageNet-1k ILSVRC-2012 dataset [19], which contains 1,281,167 training instances and 50,000test instances in 1,000 categories.
Preprocessing for training is standard and includes resizing andrandom cropping to 224 × 224 × 3, followed by a random horizontal flip.
For testing, the images areresized and center cropped.
In some of our experiments involving advanced architectures from the
Timm repository [20], we applied the test augmentations (at test time only) using its code from the
Timm repository.
Additionally, we investigated the effectiveness of TransBoost on the Food-101[21], CIFAR-10 and CIFAR-100 [22], SUN-397 [23], Stanford Cars [24], FGVC Aircraft [25], the
Describable Textures Dataset (DTD) [26] and Oxford 102 Flowers [27].
For each of these datasets,we applied the same preprocessing as in ImageNet, resulting in an image size of 224 × 224 × 3, withthe exception of CIFAR-10 and CIFAR-100, where we used random cropping with padding (croppingto 32 × 32 × 3), followed by a random horizontal flip, a single RandAugment [28] operation and
MixUp [29].
Architectures.
We examined various inductive (standard) pretrained architectures as baselinesfor TransBoost.
Specifically, we used a variety of ResNet [30] architectures: ResNet18, ResNet34,
ResNet50, ResNet101 and ResNet152, which were pretrained using the standard PyTorch trainingrecipe [31].
We also considered ResNet50-StrikesBack [32], which is the ResNet50 architecturetrained with Timm’s [20] advanced training recipe.
Additionally, we show results with more advancedarchitectures spanning various families: EfficientNetB0 [33] and MobileNetV3-L [8], ConvNext-T[7], and the Transformer models ViT-S [9] and Swin-T [34], which were pretrained with advancedprocedures of Timm’s library [20] in accordance to their original papers.
Details on the TransBoost Fine-Tuning Procedure.
Throughout all TransBoost’s experiments, wefollowed Algorithm 1, and our functions were implemented as described in Section 4.1.
We used the
LAMB optimizer [35] with a total batch size of 1024 (including 512 labeled training instances and512 unlabeled test instances) for 60 epochs, unless otherwise specified.
Following [35], our learningrate is scaled using a square-root scaling law, specifically, 8 .
25 × 10 − 6 · √ batch size.
The learningrate starts at 2 .
64 × 10 − 4 ( = 8 .
25 × 10 − 6 · √ 1024 ) with no warmup and then decayed with a cosinedecay schedule.
A weight decay of 10 − 4 is used.
The regularization hyperparameter of our loss in allour experiments was fixed to λ = 1 ; see Equation (7).
Our hyperparameters were fine-tuned based ona validation set that was sampled from the training set over ImageNet.
5.2 ImageNet Experiments: Transductive vs. Inductive
This study examines the transductive performance of TransBoost applied to various pretrainedmodels (all described in Section 5.1), and the transductive performance of the other baselines (SSLand t-FSL) versus the inductive performance of all baselines.
Table 1 presents all the results.
Wenote that some of these results were highlighted in Figure 1.
Table 1 is divided into several horizontal sections.
In the first section we present TransBoostapplied to a number of ResNet architectures.
In the second section, we discuss modern architectures,including two types of Vision Transformers, and also ConvNext-T as representative of the ConvNextfamily, which presently achieves state-of-the-art results over ImageNet [14].
Next we present the
SimCLR and SimCLRv2 SSL methods (see Appendix B for more details on how we trained themin transduction).
We note that SimCLRv2 presently dominates SSL performance over ImageNetaccording to [14].
The last table section considers two popular t-FSL methods (which are described
[Table]
[Table] [Table]
in Section 3).
The column structure of Table 1 is straightforward but the most important columnthat deserves some explanation is the Inductive / Transductive column ( 4 th column) in which wecompare the transductive performance of each experiment to the relevant inductive fully supervisedperformance.
For example, TransBoost is capable of increasing the ImageNet top-1 accuracy of
ViT-S by two percent.
Table 1 and Figure 1 demonstrate that fine-tuning with TransBoost consistently and significantlyimproves the inductive top-1 accuracy performance in all cases.
Even the tiny version of the topperforming ConvNext architecture family (presently the state-of-the-art according to [14]) is improvedby TransBoost.
Surprisingly, ViT-S performed better than ConvNext-T when taking advantage of thegiven test samples using TransBoost, indicating that members of the ViT family could be the besttransductive performers on ImageNet.
A second interesting fact is that ViT-S and Swin-T performsimilarly in the inductive setting while the ViT-S top-1 accuracy gain is improved by over x2 relative to
Swin-T. Could this advantage be related to the architecture?
Interestingly, TransBoost also improvesthe performance of ResNet50-StrikesBack, which is the standard ResNet50 architecture trainedusing a sophisticated training procedure [32] that boosts its top-1 accuracy by +4.2%.
Surprisingly,
TransBoost adds almost +1% on top of this phenomenal improvement.
Another striking relatedresult is that TransBoost improves the standard ResNet50 so that it achieves performance close to
ResNet50-StrikesBack despite the fact that our procedure is much simpler than the sophisticatedtraining procedure of [32].
Consider the first section of Table 1 (the ResNets section).
While TransBoost always improves theinductive baselines, it is evident that its relative advantage monotonically decreases with the modelsize/performance.
For example, the best transductive improvement is achieved for ResNet18 yieldinga top-1 accuracy gain of +4.15% and resulting in a top-1 accuracy of 73.91% 2 .
With respect tothe other related baselines, TransBoost consistently outperforms both the SSL and t-FSL baselinealgorithms.
Comparing the results of SimCLR to the results of SimCLRv2, we see that SimCLR outperforms SimCLRv2 in the transductive setting (they mainly differ by the additional stage ofknowledge distillation [37] that is added to SimCLRv2), which suggests that the stage of knowledgedistillation can hinder the transductive performance.
We hypothesize that distilling knowledge fromtest instances leads to overfitting of incorrect class predictions that lead to model confusion.
Finally,when evaluating the t-FSL representatives, we observe that Ent-min improves its inductive baselinewhile TIM-GD (which is an upgraded version of Ent-min) is inferior to its inductive baseline.
5.3 TransBoost’s Performance on Additional Vision Datasets
[Figure]
We now examine the performance of TransBoost across eight other datasets (all described in Section 5.1).
Throughout this study, we start with a ResNet50 architecture pretrained on ImageNet (using
PyTorch’s standard training recipe).
Then we apply inductive fine-tuning for each specific datasetand, finally, we apply our TransBoost procedure.
The results are visually depicted in the graph of Figure 3 and are also shown in Table 3 in the
Appendix.
Clearly, TransBoost outperforms its inductive baseline in all cases.
The improvement islarge and significant in all datasets with the exception of CIFAR-10 where the improvement is minor.
The most prominent result is obtained for the SUN-397 dataset, where we observe a striking +5.4%top-1 accuracy gain from 93.25% to 98.65%.
To the best of our knowledge, this is presently the bestreported result for SUN-397.
For a more in-depth discussion, we refer the reader to Appendix C.
5.4 Transductive Performance Improves with Test Set Size
[Figure]
In this study, we consider various transductive settings, where each setting is characterized by acertain training set size and a certain test set size.
We are interested in examining the final ImageNettransductive behavior of ResNet50 as a function of these sizes.
The sizes of our training sets are:5%, 10%, 20%, 100%; and for testing sets are: 10%, 25%, 50%, 100%, all taken as a fraction of theoriginal train/validation set sizes, respectively.
Consider Figure 4 showing the top-1 accuracy gains for all combinations ( 4 × 4 experiments intotal).
Importantly, we observe that whenever TransBoost uses the entire training set, its performanceincreases as the test set size grows.
This strongly indicates that TransBoost leverages the test setas a whole, a desirable property of a transductive learning algorithm.
On the other hand, and quitesurprisingly, when TransBoost only used 5% of the training set, increasing the test set size worsened its performance.
We hypothesize that this type of behavior is a result of poor prior knowledge (verysmall training set) that leads the models to wrongly analyze the relationships (similarities in our case)between pairs.
This overall bad effect increases as the test set size expands.
In addition to the observations on the effect of a sufficiently large training set, we also observethe importance of a sufficiently large test when considering the performance of TransBoost thatused 100% of the test set (marked in yellow in Figure 4).
It is evident that with this largest test set,performance monotonically increases with the training set size.
For a detailed description/discussionof this section’s experiments, see Appendix D.
5.5 Bringing the Like Together or Separating the Difference?
TransBoost is driven by Equation (2) whose informal desideratum is: revise the model so as totake apart the representations of two instances that are likely ( κ ) to belong to different classes ( δ ),which currently appear to be similar ( S ).
Rather than, or in addition to separating instances that arelikely to be different, we could consider grouping together instances that are likely to belong to thesame class.
In this section we summarize our experiments to apply TransBoost for this alternativeobjective and for a combination of both objectives.
In our study we used the ResNet50 architecture,which was pretrained on ImageNet (using PyTorch’s standard training recipe).
The transductiveperformance of TransBoost was analyzed using three variants of the transductive loss component: (1)the proposed separation component, L TransBoost ( X u |S f , δ f , κ f ) , as in Equation (7); (2) a “reciprocal”instantiation of this objective – namely, L TransBoost ( X u |√ 2 − S f , 1 − δ f , κ f ) , which brings togethersimilar instances; and (3) a variant that considers both these objectives together.
The results arepresented in Table 2.
Transductive Loss
[Table]
L [Equation] [Table] Clearly, the best performance is achieved using our proposed implementation (row
Interestingly, the reciprocal variation significantly degrades performance and the combinationof both objectives slightly reduces the top-1 accuracy.
Concluding Remarks
We presented TransBoost, a novel and powerful transductive fine-tuning procedure that can be efficiently applied on any pretrained model.
TransBoost is a deep transductive classification algorithmthat is inspired by a large margin principle such as the TSVM algorithm.
Strikingly, TransBoost consistently and significantly improves the inductive ImageNet top-1 performance of many architecturesincluding the most advanced ones.
Moreover, TransBoost is effective on a broad range of imageclassification datasets.
As described in Algorithm 1, TransBoost is a general optimization procedure that can be implementedin a range of ways depending on the similarity function ( S ), the selection function ( δ ), and theconfidence function ( κ ).
We instantiated these functions simply and straightforwardly.
A question forfuture research may be to ask whether more sophisticated choices will lead to better transductionperformance.
The ViT-S architecture achieved the best transductive classification performance inour experiments and accrued the largest gain relative to induction among advanced architectures.
Itwould be interesting to explore the question of how this Transformer architecture facilitates this gain.
In general, we can ask what is the best architecture for transductive classification.
Finally, it would bevery interesting to see whether such large performance gains are also sustained in NLP applications.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] .
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] All used models are provided at online repositories.
The code for our detection method will be provided upon acceptance.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 5.1.(c)
Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes](b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
A The Similarity Function is Bounded
In the following, we claim and prove that S f is bounded.
Claim 1.
Let f : X → { 1 , . . .
C } be a classification model and x 1 , x 2 ∈ X input instances, where
C is the number of classes.
Accordingly, our implementation of TransBoost’s symmetric pairwisesimilarity function, S f (4), satisfies the following:
[Equation]
Proof.
Recalling our implementation of S f (4),
[Equation]
where ˆ p ( x | f ) is the empirical class probabilities of f applied on x .
Since || · || is non-negative,we obtain S f ( x 1 , x 2 ) ≤ √ 2 .
Now we need to show that S f ( x 1 , x 2 ) ≥ 0 , which holds iff,
[Equation]
[Equation]
Considering the left-hand side of the inequality,
[Equation]
Following a sub case of Hölder inequality, ∀ v ∈ R n ; || v || 2 ≤
|| v || 1 .
Therefore, using the lastequation we obtain:
[Equation]
[Equation]
p [Equation]
p
[Equation]
=
[Equation] (8) ≤ [Equation] Where (8, 9) hold since ˆ p is a probability vector.
Hence for any x ∈ X , ∥ ˆ p ( x | f ) ∥ 1 =(cid:80) Ci =1 | ˆ p i ( x | f ) | = 1 and ∀ 1 ≤ i ≤ C ; 0 ≤ ˆ p i ( x | f ) ≤ 1 .
■
B Transduction Training Details of SimCLR and SimCLRv2
The SimCLR [11] algorithm consists of self-supervised pretraining and supervised fine-tuning.
SimCLRv2 [12] incorporates approximately the same steps, but adds a final knowledge distillation[37] stage that has been found powerful in inductive SSL (see [12] for further details).
SimCLR (and SimCLRv2), however, are designed for SSL rather than transduction; therfore, we willnow elaborate how they were trained for transduction.
As a first point, we note that both SimCLRand SimCLRv2 were applied to the ResNet50 and ResNet152 architectures using pretrained weightsprovided by the authors.
Next, we fine-tuned the model, using both the labeled training set as well asthe unlabeled test set, for another 100 epochs.
We followed the pretraining scheme as suggested bythe authors.
In the second step of supervised fine-tuning, we followed the SimCLRv2 instructionsgiven by the authors.
For the final stage (self-distillation), SimCLRv2 optimizes a weighted lossfunction based on labeled training instances applied to standard CE and unlabeled training instancesapplied to the CE distillation loss.
In our case, the unlabeled set is the transductive set.
C Supplement for the Additional Vision Datasets Experiments
Here we discuss the details in the experiments on TransBoost using the additional vision datasets thatwere introduced in Section 5.3, as well as an additional observation offered in Appendix C.1.
Table 3compares the TransBoost performance to the standard inductive (fully supervised) performance andhighlights some relevant properties of the datasets we used.
[Table]
C.1 Transductive Performance Gain Improves as the Number of Classes Grows
Table 3 displays an interesting pattern regarding the number of classes.
For experiments conductedusing both non-CIFAR and CIFAR groups, we observe that TransBoost performs better on datasetswith many classes than on datasets with a few classes.
Specifically, among non-CIFAR experiments,it can be seen that the highest top-1 accuracy gain is achieved on SUN-397 while the experimenton DTD attained the smallest improvement.
SUN-397 and DTD have the largest and the smallestnumber of classes, respectively, among non-CIFAR datasets.
This effect is also observed when the
CIFAR datasets are used.
On the basis of these observations, we hypothesize that the more classesthe dataset has, the more likely similar representations, which belong to different classes, will befound and, therefore, there are more target test instances to improve.
D Supplement for the ImageNet Subsets Experiments
This section provides a supplementary in-depth discussion of our experiments on ImageNet trainingand test subsets, which are varying in their sizes (introduced in Section 5.4).
In this study, weused the ResNet50 architecture that was pretrained on ImageNet combinations of training and testsubsets (using PyTorch’s standard training recipe).
For experiments that do not utilize the entiretraining set, we applied the TransBoost procedure with a batch size of 128.
Although TransBoost wasoptimized for a specific test set, here we evaluate its performance in various inductive settings, whereunseen test instances are examined at test time.
Additionally, we present a comprehensive analysisof TransBoost’s performance in various transductive settings as well as a standard inductive (fullysupervised) performance analysis as a baseline.
The complete results are presented in Table 4.
Table 4 presents 16 experiments of TransBoost’s procedure performed on all combinations of the
ImageNet training set fractions: 5%, 10%, 20%, 100%; and the ImageNet test set fractions: 10%,25%, 50%, 100%.
The table’s sections represent training set fractions, and the rows in each sectionrepresent test set fractions.
There are two main column groups highlighted in green and blue.
Green columns present comprehensive results in transductive settings, whereas blue columns presentcomprehensive results in inductive settings.
Across the two settings (transductive / inductive),
TransBoost’s performance is compared to the standard inductive (fully supervised) performance.
Forinstance, using 20% of the training set and 25% of the test set, we obtained the best top-1 accuracygain of +3.58% in the transductive setting while the performance in the inductive setting degraded by-1.21%.
We note that the performance in the inductive setting (highlighted in blue) is evaluated usingthe test instances that were not used at training time.
As can be seen, TransBoost consistently outperforms the baselines in transductive settings whilethe baselines consistently outperform TransBoost in inductive settings.
This behavior is indeedwhat one may expect from real transductive models, which will only be asked for the given test set.
Interestingly, TransBoost’s best top-1 accuracy gain – an impressive improvement of +3.58% – was
ImageNetFraction /
InstancesPerClass
[Table]
achieved by the experiment that used 20% of the training set and 25% of the test set.
Our best top-1accuracy (79.11%) was attained in the experiment that used the entire training and test sets.
Forfurther details, we refer the reader to Appendix D.1 and Appendix D.2, below.
D.1 TransBoost’s Inductive Performance Increases with Test Set Size
In Figure 5, we present an interesting trend of TransBoost’s top-1 accuracy gains in various inductivesettings (derived from Table 4).
Note that the fractions described in the figure legend refer tothe test instances that participated in the training procedure, while the inductive performance wasexamined using the rest of the instances.
We have already discussed the fact that all inductivebaselines outperform TransBoost in all inductive settings.
This can be expected from a transductivemodel that specializes in a particular test set.
Figure 5, however, reveals an interesting patternwhen the entire training set is not used: the more test instances are provided at training time, thebetter TransBoost’s performance in the inductive setting.
This observation implies that our proposedoptimization procedure learns better to generalize unseen instances when the ratio of test instances(out of the training and test instances combined) is significant.
Here we provide an additional analysis of the relative improvement in our proposed loss component,
L TransBoost (see Section 4.1), across the experiments that we described above.
Specifically, we approximated TransBoost’s loss following Algorithm 1 (the approximation stage only) on the inductivebaselines as well as on TransBoost’s output models in the transductive setting.
Then, we calculatedtheir relative improvements in (%) and presented the results in Figure 6.
For example, our bestrelative improvement in L TransBoost was given by the experiment that used the entire training and testsets with a relative improvement of +99.3%.
As shown in Figure 6, the larger the training and/or test sets TransBoost uses, the larger is theimprovement of the loss L TransBoost .
This suggests that the greater the prior knowledge of the modelis, the easier it is to optimize L TransBoost .
Moreover, the more test instances we use in the TransBoostoptimization procedure, the more L TransBoost does improve.
We contend that this is how a transductiveprocedure should behave, since the test instances are used as a group rather than treating test instancesindependently.