Pre-Trained Model Reusability Evaluation for Small-Data Transfer Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We study model reusability evaluation (MRE) for source pre-trained models: evaluating their transfer learning performance to new target tasks.
In special, we focuson the setting under which the target training datasets are small, making it difﬁcult to produce reliable MRE scores using them.
Under this situation, we proposesynergistic learning for building the task-model metric, which can be realized bycollecting a set of pre-trained models and asking a group of data providers to participate.
We provide theoretical guarantees to show that the learned task-modelmetric distances can serve as trustworthy MRE scores, and propose synergisticlearning algorithms and models for general learning tasks.
Experiments show thatthe MRE models learned by synergistic learning can generate signiﬁcantly morereliable MRE scores than existing approaches for small-data transfer learning
Introduction
When selecting a model from a big model pool for doing model transfer, one has to do modelreusability evaluation (MRE) ﬁrst: evaluating the transfer learning performance of the models to thetarget task and identifying the best model.
The role of MRE is crucial since no matter how good thetransfer learning strategy is, incorrect MRE would still lead to the danger of negative transfer.
MREhas received growing attention in recent years [Achille et al., 2019, Tran et al., 2019, Nguyen et al.
,2020, Wu et al., 2020, Ding and Zhou, 2020, You et al., 2021].
But most existing studies focus onlarge-data MRE, under which the target dataset is sufﬁciently large.
In this work, we focus on small-data MRE, under which the target training datasets are small.
Reusing pre-trained models is essential under the small-data scenario since learning from scratchis difﬁcult.
Unfortunately, large-data MRE approaches are usually invalid for small data since theyusually focus more on simplicity and efﬁciency, but not generalization and robustness, which areessential for small-data MRE.
It is indeed challenging to obtain reliable MRE results under thesmall-data scenario due to the fundamental burden set by the laws of statistics.
Our solution is based on two observations.
First, model reusability usually depends on the taskmodel relationship other than the task-independent property of any speciﬁc small sample.
We couldgrasp this relationship by meta-learning an MRE function among tasks and models.
Second, thislearning process is often realizable in practice, in special for large model zoo platforms where sufﬁcient pre-trained models and data providers are available.
Based on these observations, we conductthe following studies in this paper:
Problem formuation.
We provide the formulation of small-data MRE and synergistic learning tolearn the MRE function by metric learning.
Synergisitc learning works for general learning scenarios beyond classiﬁcation, on which most previous MRE approaches focus.
Theoretical analysis.
We propose access risk analysis showing that the MRE model learned by syn- ergistic learning guarantees to generate reliable MRE scores.
The theory not only provides guarantee
[Table]
SynergisticLearning
[Table]
[Figure]
even for using non-convex deep networks as the MRE model, but also motivates feature pre-trainingin synergistic learning.
Algorithm and model design.
We propose synergistic learning algorithms and MRE model structures for general learning tasks, which have the auxiliary advantage of protecting data privacy.
Meanwhile, we propose a more elaborate MRE model strucuture for classiﬁcation.
Experimental veriﬁcation.
Experimental results show that synergistic learning can generate significantly more reliable MRE scores for small-data transfer learning than existing MRE approaches.
Problem Setup
In this section, we provide the formal deﬁnition of small-data MRE and synergistic learning.
2.1 Small-Data MRE
Denote by Z = X
× Y the observation space, in which X , Y are the input and output spacesrespectively.
A target learning task µ can be represented by a probability measure 1 µ ( x, y )
oversome Z µ ⊆ Z .
Furthermore, we assume that all µ are drawn from the task environment T , a 49 probability measure over the space of all target tasks.
In small-data MRE, target tasks µ are drawn from T .
For each µ , the objective is to learn a predictionrule f µ such that for any observation ( x, y ) ∼ µ , f µ ( x ) is close to y .
The learner of µ will be givena target training dataset S µ , a set of pre-trained models H , a model transfer strategy π and an MREfunction v π .
S µ = { z k = ( x k , y k ) }
Kk =1 is a sample drawn from µ such that the sample size Kis a small number.
H is the pre-trained model set in which each model h ∈ H is drawn fromthe model environment M , a probability measure over the space of all pre-trained models.
Weassume that the models in H are sampled independently from µ while different H can be givenfor different µ .
π ( S µ , h ) is a transfer learning strategy which can be used to transfer a pre-trainedmodel h into f µ using S µ .
We do not restrict the choice of π , but assume that π is ﬁxed for anytarget task.
The MRE function v π ( S µ , h ) is a real-valued function taking S µ and h as the inputsand outputing the transferability score.
Below we use v instead of v π to simplify notations.
Withoutloss of generality, among all possible v , we deﬁne v ∗ as the optimal MRE function such that for anyµ, S µ , h 1 , h 2 , v ∗ ( S µ , h 1 ) < v ∗ ( S µ , h 2 )
if and only if h 1 has better transfer learning performancewith S µ than h 2 2 If no special properties exist for v ∗ , estimating v ∗ would be extremely difﬁcult forsmall-data MRE.
Fortunately, in general, we could assume that model reusability usually dependson the task-model relationship other than the task-independent property of any speciﬁc small sample,as discussed in Section 1.
Therefore, we assume that the ground-truth model transferability dependsonly on the target task µ but does not depend on the random draw of S µ from µ , i.e. ∀ S µ , S " µ ∼µ, v ∗ ( S µ ) = v ∗ ( S " µ ) .
Thus we can use the notation v ∗ ( µ, h ) instead of v ∗ ( S µ , h ) for any S µ , h .
How this assumption can be relaxed is discussed in Section 7.
2.2 Synergistic Learning
Synergistic learning is the preparation stage for learning an MRE function ˆ v that accurately approximate v ∗ before any MRE problem deﬁned in Section 2.1 starts.
A synergistic learning process workswith the following three prerequisites:
• A set of pre-trained models H SynLearn
= { h k } N m k
=1 drawn from M are given.
Note that themodels in this set could be different from models given in any future MRE problems;• A set of data providers
D =
{Z d } Dd =1 participate in learning, such that each data providerrepresents an observation subspace Z d ⊆ Z .
Furthermore, ∪ Z d = Z .
We deﬁne theclosure of any task µ ∼ T as ¯ Z µ =
min L {Z d 1 , Z d 2 , . . .
Z d L } s.t.
Z µ ⊆ ∪ Ll =1 Z d l .•
A validation MRE function v valid is provided for synergistic learning.
v valid is unbiasedbetween any task and model 3 : ∀ µ, h, v valid ( µ, h ) =
E S µ ∼ µ [ v valid ( S µ , h )]
= v ∗ ( µ, h ) .
The basic idea of synergistic learning is to establish a metric ˆ v ( S µ , h ) between the target trainingdatasets and the models so that the metric distance could be used as the MRE score.
In Section 4,we will show that learning ˆ v only requires the data providers to answer two kinds of queries:
• SQUERY
( ψ s , Z d , µ, M ) : a single data provider Z d is queried to sample M observationsfrom a given probability measure µ over Z d , and return the output of ψ s ( { z i } M i =1 ) , a function of the observations.
• MQUERY ( ψ m , {Z d i } Qi =1 , { µ i } Qi =1 , { M i } Qi =1 ) : multiple data providers {Z d i } Qi =1 are queriedjointly.
Each data provider Z d i sample M i observations from a given probability measure
µ i over Z d i .
Aggregating the sampled observations from all data providers, the outputof ψ m ( { z 1 ,j } M 1 j =1 , { z 2 ,j } M 2 j =1 , . .
. , { z Q,j } M Q j =1 ) , a function of all observations, is returned.
Furthermore, ψ m cannot be realized by aggregating multiple SQUERY .
The main difference between these two queries lies in the scope of involved data providers, which isimportant when the data privacy is sensitive.
SQUERY is answered by a single data provider.
How thiskind of queries can be answered in privacy-guaranteed ways has received many studies [Koneˇcn ýet al., 2016, Zinkevich et al., 2010].
In contrast, MQUERY needs to be answered by multiple dataproviders jointly.
Protecting data privacy is much harder in this situation since their data need to beaggregated.
Therefore, we set an auxiliary target of reducing the number of times to use MQUERY .
Finally, for learning ˆ v , we assume that ˆ v is formed by three parts: (1) g µ ( S µ ) , the task featurebackbone; (2) g h ( h ) , the combination of the model speciﬁcation generator and the model featurebackbone; (3) d θ ( c µ , c h ) , the metric module, in which c µ is the output of g µ ( S µ ) , c h is the output ofg h ( h ) , and θ is the learnable parameter of d θ ( c µ , c h ) .
g µ and g h transfer target training dataset S µand pre-trained model h into their representations c µ and c h .
The metric module then calculates themetric distance between c µ and c h as the MRE score.
The details are introduced in Section 4.2.
Theoretical Analysis
In this section, we discuss the theoretical foundation of synergistic learning.
Readers who are interested more on algorithmic ideas can skip this section without affecting understanding signiﬁcantly.
In the analysis, we assume that g µ and g h are ﬁxed and focus on learning the metric module.
Thusthe learnable parameter for ˆ v is θ .
For metric-based synergistic learning, we assume that N m pre-trained models { h k } N m k
=1 sampled from M , N t tasks { µ i } N t i =1 sampled from T and N S target training datasets { S µ i ,j } N S
j =1sampled from each task µ i are used for training.
We also assume that for any pair of h iand S µ j ,k , the ground-truth MRE score v ∗ ( µ i , h k ) are given.
Deﬁne r v ∗ ( S µ , S µ " , h, h " ; ˆ v ) as
I [ ∆ v ∗ ( µ, µ " , h, h " ) < 0] I [ ∆ ˆ v ( S µ , S µ " , h, h " ) ≥ 0] in which ∆ v ∗ ( µ, µ " , h, h " )
= v ∗ ( µ, h ) −v ∗ ( µ " , h " ) , ∆ ˆ v ( S µ , S µ " , h, h " )
=
ˆ v ( S µ , h ) − ˆ v ( S µ " , h " ) , and I is the indicator function.
Since v ∗is the ground-truth MRE function, a desirable ˆ v should minimize
[Equation]
where the expectation is taken over S µ ∼ µ, S µ " ∼ µ " , µ, µ " ∼ T , h, h " ∼ M .
While for the 117 convenience of optimization and analysis, we deﬁne our objective using the triplet surrogate loss.
Deﬁne r γ v ∗ ( S µ , S µ " , h, h " ; ˆ v ) as I [ ∆ v ∗ ( µ, µ " , h, h " ) < 0]
[ ∆ ˆ v ( S µ , S µ " , h, h " ) + γ ] + in which [ x ] + =max { x, 0 } and γ > 0 .
When γ > 1 , r γ v ∗ upper bounds r v ∗ .
Meanwhile, r γ v ∗ is consistent with r v ∗since r γ v ∗ ( S µ , S µ " , h, h " ; ˆ v ) = 0 indicates that r v ∗ ( S µ , S µ " , h, h " ; ˆ v ) = 0 .
Therefore, we deﬁne ourgoal as minimizing the following expected risk
[Equation]
A possible way to achieve this goal is to minimize the following empirical risk
[Equation]
where the summation is taken over all models, tasks and datasets used for training.
We want to know whether minimizing the empirical risk ˆ R γ (ˆ v ) indeed leads to the minimization of the expected risk
R γ (ˆ v ) .
Denote by Θ the parameter space for θ and use ˆ v θ to denote the ˆ v with learnable parameter θ .
We provide an access risk bound showing the effectiveness of minimizing the above empirical risk.
Use ¯ G θ ( S µ , S µ " , h, h " ) to denote [ ∆ ˆ v θ ( S µ , S µ " , h, h " )
+ γ ] + .
Additionally, we assume that ∇ [ ¯ G θ ( S µ , S µ " , h, h " )]
= 0 when ¯ G θ ( S µ , S µ " , h, h " ) = 0 .
Furthermore, let
[Equation]
in which the summation is taken over all tasks, datasets and models used for training and wehave ∇ − [ S µ , S µ " , h, h " ] = I [ ∆ v ∗ ( µ, µ " , h, h " ) < 0] ∇ [ ¯ G θ ( S µ , S µ " , h, h " )] .
Meanwhile, we denote
V = max i ∈ [ N t ] E S µi ∼ µ i [ c 2 ( S µ i )] .
Now we are ready to state the following theorem whose proof is 126 provided in the appendix (Section A).
Theorem 3.1.
Under Assumption A.1, there exist optimal parameter θ ∗ ∈ Θ and constant β > 0 ,∀ θ ∈ Θ , the following event
[Equation]
" holds with high probability, in which ∆ ( N t , N m , N S ) =
˜ O
(1 / √ N t , 1 / √ N m , V/ √ N S , 1 /N S ) .
Theorem 3.1 shows several interesting insights.
First, ∇ [ ˆ R ( ¯ G θ )] is the gradient norm of the metricdistance gaps from the training data.
The bound shows that the access risk will be small whenthe gradient norm tends to zero, even when the global minimum of the empirical risk has not beenreached.
This makes the bound informative when non-convex models, such as DNNs, are used.
Second, the bound has the normal O (1 / √ N t ) , O (1 / √ N m ) sample complexity dependence for bothtasks and models.
This is in agreement with our intuition such that lacking any of the tasks andmodels would lead to the failure of learning.
Finally, the most interesting take-away is the samplecomplexity for target datasets in each task, which has a dependence of the feature variance V .
If Vis small, the order becomes O (1 /N S ) instead of O (1 / √ N S ) , a signiﬁcant drop for the number ofthe training datasets.
In Section 4, we show that this result inspires us to include feature variancereduction in the synergistic learning process, which would signiﬁcantly reduce using MQUERY .
Learning Method
In this section, we introduce the synergistic learning algorithm and the MRE model.
4.1 Synergistic Learning Algorithm
The realization of synergistic learning is illustrated in Algorithm 1.
In this section, besides the metriclearning step of minimizing Equation 2 (Line 11), we discuss other crucial steps below in general.
We consider two general synergistic learning settings.
One is named isolated closure setting , whichindicates that any task closure includes a single data provider.
Otherwise, we name it groupedclosure setting , which indicates that there exist multi-data-provider task closures.
Task and model sampling (Line 8).
To generate the training data for synergistic learning, tasksand models are needed be sampled.
At this stage, no interaction to the data providers is needed.
M ; 9:
←
, v ∗ µ i h k ) ← COMBINE [ { ψ µ i , Z d , h k
} | ¯ Z µi | d =1 ] , i ∈ t
, k ∈ m ] , d ∈
¯ Z µ i
Target Dataset Generation : S µ i ← MQUERY
Z µ i i =
t ; 11: v ; { ti mk v i h
} t,mi k =1 ) to minimize
2; 12: until reach end.
Reusability validation (Line 9).
The objective for reusability validation is to acquire value of v ∗between any pairs of task and model that are sampled.
In general, this can be done by sendingmodels to the data providers and estimate v ∗ using v valid .
Under the isolated closure setting, only
SQUERY is needed obviously.
We could ask the data providers to use sufﬁciently large data, makingv valid an accurate estimator.
But the key challenge appears under the grouped closure setting sincethe statistics used for calculating v valid should be obtained from multiple providers at the sametime.
To achieve this by SQUERY , we require v valid to have a special structure.
Speciﬁcally, fortask µ , let S µ ( Z d ) be a sample generated from the marginal distribution of µ over Z d , a member ofits task closure.
We require v valid ( µ, h ) =
COMBINE [ { ψ ( µ, Z d , h ) } | ¯ Z µ | d =1 ] , in which ψ ( µ, Z d , h ) =
E S µ ( Z d ) ∼ µ ( Z d ) [ ψ ( S µ ( Z d ) , h )] .
In this construction, ψ ( S µ ( Z d ) , h ) is a function over single data 162 providers and COMBINE is an aggregation function decided by speciﬁc MRE function, thus only
SQUERY is needed for calculation.
Target dataset generation (Line 10).
For isolated closure setting, this step can be done by samplingfrom a data provider using SQUERY .
However, for grouped closure setting, MQUERY is necessary forthis step since the target datasets must include the raw data.
This issue can be relieved by the insightbrought from Theorem 3.1: For each task, the number of the target data needed is closely related tothe data variance.
A feature pre-training stage can be introduced to reduce using MQUERY .
Feature pre-training (Line 2-6).
Feature pre-training is the preparation stage for synergistic learning.
It is participated by all data providers, aiming at learning a feature extractor g µ which couldoutput low-variance features for the raw data.
Once g µ is learned, it would be integrated into the
MRE model.
As a result, in synergstic learning, the data providers only need to use much fewerdata for answering MQUERY .
On the other hand, the feature pre-training stage itself should be doneby using only SQUERY .
We argue that this can be realized by using learning objectives which are decomposable w.r.t.
the data providers .
More speciﬁcally, a decomposable objective is the summationof individual objectives, such that each individual objective L ddecom only takes the data from a singledata provider Z d as its input.
Then the purpose of the queries reduces to calculating each L ddecom ,which can be done by SQUERY as expected.
Due to space limitation, we provide more details on how to implement the above synergistic learningsteps to different learning scenarios in Section B.
4.2 MRE Model
First, we introduce the basic structure of the MRE model deﬁned in Section 3, which is illustratedin (a) of Figure 2.
In this model, the target data is ﬁrst processed by the task feature backbone togenerate the feature c µ .
Afterwards, a task transform module is responsible to generate the taskrepresentation φ µ which aggregates all the information from the target dataset.
For isolated closuresetting, the task transform module can be simply an instance-wise averaging operation.
In correspondence, before the model feature backbone, a model speciﬁcation generator is introduced whichcan be any function that transforms a model into a vector.
The model feature c h is then transformed
[Table]
[Table]
LinearTransform
[Table]
+CkKVjjRoqv69SLDQeiz8dFNgM9SLXib+53VjE1x7CZNRbKgks6Ag5siEKCsA9ZmixPBxSjBRLP0VkSFWmJi0prkUX2SdOIsNLJNWrepcVmv355X6Td5OEY7gGE7BgSuowx00oAkEIniBV3iznq1368P6nK0WrPzmEOZgff0CdsCXWg==</latexit> w 1 ,
[Figure]
into the model representation φ h using the model transform module.
For isolated closure setting, themodel transform module can usually be omitted, i.e. c h = φ h .
For many learning scenarios, using more elaborate MRE model structure is useful, such as classiﬁcation.
Below we consider small-data transfer learning for classiﬁcation with supervised pre-trainedmodels.
A target task µ has L T classes { y l } L T l =1 to distinguish, in which all the classes belong to thetask class universe Y T .
Meanwhile, for a pre-trained L M -way classiﬁcation model h , we denote by{ ˆ y l " } L M l " =1 the classes that the models are trained on.
We assume that all the model classes belong 196 to the model class universe
Y M , which could be different from Y T .
Similar to [Tran et al., 2019,
Nguyen et al., 2020, You et al., 2021], we assume that any transfer strategy for this scenario, suchas global ﬁne-tuning and head re-training, can be used.
While according to our experience, head retraining, in which the feature backbones of the pre-trained models are frozen and only the predictionheads are re-trained, is more proper than global ﬁne-tuning for small-data transfer learning.
The overall structure of the MRE model for classiﬁcation follows the basic model, but the task andmodel transform modules have more complex structures.
For the task transform module ((b) of
Figure 2), we take care of modeling the relationship among task classes.
On one hand, instead ofdirectly generating the feature representation of the whole task φ µ , the module generates the features{ φ lµ } L T l =1 for all task classes ﬁrst, and then aggregate them to form φ µ .
Any set-to-set transformmodule, such as Deep Sets [Zaheer et al., 2017] and Transformer [Vaswani et al., 2017, Ye et al.
,2020], can be used.
What is essential here is to generate the feature for one task class based on thecontext information from all other task classes.
More importantly, we propose an attention-based model transform module.
This module is motivated by our observations during studying MRE: there is a close connection between p ( y l , ˆ y l " ) andmodel reusability for classiﬁcation, which is also pointed out by recent studies [Tran et al., 2019,
Nguyen et al., 2020].
The module is illustrated in (c) of Figure 2.
First, the module takes both themodel and task class features { c l " h } L M l "
=1 , { c lµ } L T l =1 as its inputs and transform them into the modelattention-aware task class features { φ lh } L T l =1 .
Subsequently, { φ lh } L T l =1 are aggregated to form the 215 model feature φ h .
The core component of this module is the model ensembler ((d) of Figure 2).
Fora pair of model and task classes
y l , ˆ y l " , the model ensembler calculates the attention weight of y l toˆ y l " , which is w l,l " , with the inner-product attention.
And then, the model class features are linearlycombined by the attention weights to form φ lh which represents the selected model class informationon y l .
We require w l,l " to be closely related to p ( y l , ˆ y l " ) to make it represent the correlation betweeny l and ˆ y l " .
Therefore, besides the metric loss deﬁned in Equation 2, we introduce an additional attention supervision loss to supervise the learning of attention weights.
For w l,l " , we treat it as the outputprobability of a binary classiﬁer.
The training labels are generated from p ( y l , ˆ y l " ) .
To be speciﬁc,we set the label to be one if p ( y l | ˆ y l " ) > γ 1 , p (ˆ y l " | y l ) > γ 2 and zero otherwise, in which γ 1 , γ 2 aretwo thresholds.
And then, the attention supervision loss is calculated from the logistic loss.
Finally,the overall training loss is formulated as
[Equation]
) in which L metric is the small-batch version of the metric loss deﬁned in Equation 2, L att is theattention supervision loss deﬁned above, and w att is the weight for L att .
[Figure] [Figure]
Experiments
In the experiments, we ﬁrst do metric visualization to verify whether synergistic learning can learnmeaningful metric space.
Furthermore, we conduct experiments for both in-dataset and cross-dataset
MRE to verify the performance of synergistic learning.
All the experiments are conducted on serverswith NVIDIA Tesla V100 GPUs.
The code is implemented using Tensorﬂow [Abadi et al., 2016]under Apache License 2.0 and is available on http://xxx.xxx.xxx.
More details of the experimentalsetups are discussed in Section C and more experimental results are included in Section D.
5.1 Metric Visualization
We adopt two ten-class datasets MNIST [LeCun et al., 1998] and CIFAR-10 [Krizhevsky, 2009] tovisualize the task-model metric learned by synergistic learning.
For each dataset, we randomly generated 20 ﬁve-class pre-trained models, as well as 20 data providers with the same class assignments,for synergistic learning, and another 20 models for testing.
We treat each model as ﬁve detectorsfor the corresponding classes and try to use synergistic learning to obtain the detector-data metric.
We consider two settings of synergistic learning.
The ﬁrst is learning with full data: all training setsare used for metric learning and there is no feature pre-training stage.
The second is learning withpart data: 10% of the training set for each class is used for metric learning, and there is a featurepre-training stage involved using the full training set.
Figure 3 illustrates the t-SNE [Maaten and
Hinton, 2008] visualizations of the learned metric, which is calculated from the testing models andinstances.
We can see that meaningful metric distance spaces are learned for both learning with fulland part data: synergistic learning makes the instances and detectors with the same classes closer.
These results provide preliminary proofs for the effectiveness of synergistic learning.
[Table]
5.2 In-Dataset MRE
Next, we use the dSprites dataset [Matthey et al., 2017] for testing in-dataset MRE.
dSprites consistsof images generated from six latent factors.
We select three factors, shape, scale, orientation, toform the task domains.
Another two real-valued factors, position X and Y, are used as the predictiontargets.
There are 720 domains and 1024 instances under each domain in total.
For each domain, werandomly select 800 instances as the training set which are used to obtain 720 pre-trained models.
The remaining 224 instances are used as the testing set.
We randomly select 503 domains as thein-distribution domains and the remaining 217 domains as the out-distribution domains.
Under thissetting, we treat each in-distribution domain as a data provider, thus there are 503 data providersin total.
For synergistic learning, we only use the training sets and models from the in-distributiondomains.
v valid ( µ, h ) is set according to the mean squared error (MSE).
We set training K = 10 ,but testing K = 1 for verifying the performance under the extremely challenging situation.
Duringtesting, we generate testing tasks using the testing sets from either the in-distribution domains or theout-distribution domains.
For in-distribution domains, the models used for testing consist of onlyin-distribution models.
The out-distribution domains follow the similar rule.
Figure 4 shows theranking performance comparison between using MSE directly on the target datasets and synergisticlearning (SynLearn).
It can be observed that the synergistic learning ourperforms the naive MSEprediction signiﬁcantly: under the extreme situation where testing K = 1 , MSE almost fails totallyin generating useful rankings, while synergistic learning can still perform desirably.
5.3 Cross-Dataset MRE
Finally, we use CIFAR-100 [Krizhevsky, 2009] and MiniImageNet [Vinyals et al., 2016] for crossdataset MRE experiments.
Results on more datasets are included in Section D.
We consider twosettings for experiments: Reuse CIFAR-100 pre-trained models on MiniImageNet target tasks andthe opposite.
For each setting, we pre-train 200 20-class models on the source dataset, in which100 models are used for synergistic learning and the other 100 for testing.
We use head re-trainingas the transfer strategy.
For the target dataset, its pre-deﬁned training set is used for synergisticlearning and its pre-deﬁned testing set is used for testing.
All training and testing tasks are ﬁxedto be ﬁve-way classiﬁcation.
For synergistic learning, we randomly generate 100 data providers,each of which holds the data of ﬁve target dataset classes.
Full training set is used for metric pretraining and 10% of the training set is used for metric learning.
For each testing task, 50 instancesare sampled from each class to test accuracy.
For performance evaluation, we use Kendall’s τcoefﬁcient [Kendall, 1938] to measure the rank correlation between the MRE scores and the testingaccuracy.
For emphasizing the performance on top-performed models, we also employ the weightedversion of Kendall’s τ -coefﬁcient, τ w , for which an exchange between elements with rank r and s(starting from zero) has weight 1 / ( r +1)+1 / ( s +1)
[Vigna, 2015].
We compare the performance ofsynergistic learning (SynLearn) with three state-of-the-art MRE methods: NCE
[Tran et al., 2019],
LEEP [Nguyen et al., 2020] and LogME
[You et al., 2021].
The results are illustrated in Table1.
It can be observed that synergistic learning signiﬁcantly outperforms other methods for smalldata MRE.
Note that for results in Table 1, we ﬁx K = 10 during synergistic learning, while weobserve signiﬁcantly more robust performance of SynLearn over other approaches when the testing
K varies.
We conduct ablation studies to verify the effectiveness of the attention supervision, thefeature pre-training, and the choice of the training K .
The results are provided in Section D.
Related Work
Direct approaches for MRE.
The direct approaches are based on the statistics calculated on thetarget training datasets as the MRE scores.
The direct approaches involve only simple statistics calculation, meanwhile no auxiliary information is used, thus are usually quite simple and efﬁcient.
Therepresentative studies of the direct approaches are [Tran et al., 2019, Nguyen et al., 2020].
In [Tranet al., 2019], the negative conditional entropy (NCE) score is proposed, which is an informationtheoretic quantity measuring the entropy for p ( y | ˆ y ) .
In [Nguyen et al., 2020], the log expectedempirical prediction (LEEP) score is proposed, which can be regarded as an improvement of NCE.
The LEEP score also closely related to p ( y | ˆ y ) .
But it uses the soft prediction probability in calculation, to take the place of the hard label assignment calculation in NCE.
Thus LEEP uses moreinformation of prediction uncertainty.
However, as shown in our experiments, both NCE and LEEPsuffer from signiﬁcant performance degeneration for small-data MRE.
This is not surprising sincethe statistics calculated from small data usually have higher variance.
Speciﬁcation-based approaches for MRE.
The speciﬁcation-based approaches use side information of the models, the model speciﬁcations , to identify model reusability [Wu et al., 2020, Dingand Zhou, 2020].
Wu et al.
[2020] proposes the RKME speciﬁcations, which are data prototypesgenerated by kernel mean embedding.
The RKME speciﬁcations reﬂect the density of the trainingdomains for the pre-trained models, which can be used to calculate the distance between source andtarget domains.
In [Ding and Zhou, 2020], the reusability indicator speciﬁcation is proposed, whichcan be used to detect in-task-domain instances.
The model speciﬁcations are designed to take auxiliary information for MRE, which can not be extracted from the model and the target data directly.
Thus the speciﬁcation-based approaches could beneﬁt from the additional side information.
However, the existing speciﬁcation-based approaches are designed for the transductive learning setting,under which the testing data of the target task are given during MRE.
This is different from theinductive learning setting studied in this paper.
Learning-based approaches for MRE.
The learning-based approaches conducts learning for MRE.
Similar to the existing speciﬁcation-based approaches, the testing-stage models and tasks are usedfor learning.
In comparison, no testing tasks and models are necessary for synergistic learning.
Therepresentative approaches are [You et al., 2021, Achille et al., 2019].
In [You et al., 2021], the logarithm of maximum evidence (LogME) approach is proposed.
Different from NCE and LEEP whichaim at ﬁnding the correlation between the source model predictions and the target outputs, LogMEbuilds the correlation between source model features and the target outputs.
Training using targetdata is necessary for LogME.
The advantage of LogME is its wide applicability for different learning problems.
While its performance degenerates more signiﬁcantly than NCE and LEEP underthe small-data scenario.
This is likely caused by the necessity of learning on the target data whichwould lead to stronger over-ﬁtting.
In [Achille et al., 2019], the Model2Vec approach is proposed.
Model2Vec uses metric learning to build task-model metric, which is similar to synergistic learning.
But Model2Vec focuses on generating the metric distances for a ﬁxed set of models, thus isa learning-based approach for MRE.
In comparison, synergistic learning generates the task-modelmetric for future MRE problems in which the models are unseen during synergistic learning.
Few-shot learning.
In few-shot learning (FSL) [Vinyals et al., 2016, Ha et al., 2017, Snell et al.
,2017, Finn et al., 2017], a meta-model is learned to solve future small-data learning tasks.
Usually,
FSL does not assume to use pre-trained models except for a few cases [Chowdhury et al., 2021].
For MRE, its relationship to FSL is similar to that to transfer learning: MRE can serve as a goodpreparation step for any FSL task that is allowed to use a pool of pre-trained models.
Limitations and Future Work
In this work, we proposed the synergistic learning approach for the small-data model reusabilityevaluation (MRE) problem.
This approach is based on the assumption that the model transferabilityhas little relationship with the task-independent properties of the small sample.
But in real problems,bad samples indeed exist whose task-independent properties affect transfer performance.
We believethat fundamental limits exist for dealing with these samples, thus the relaxation of this assumption ischallenging.
Note that Theorem 3.1 does not rely on this assumption, thus synergistic learning canbe done even when it is not held.
While lacking this special property could possibly degenerate theperformance.
Exploring how to tackle this challenge would be an important future research topic.