Pre-Trained Model Reusability Evaluation for Small-Data Transfer Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We study model reusability evaluation (MRE) for source pre-trained models: evaluating their transfer learning performance to new target tasks.
In special, we focuson the setting under which the target training datasets are small, making it difﬁcult to produce reliable MRE scores using them.
Under this situation, we proposesynergistic learning for building the task-model metric, which can be realized bycollecting a set of pre-trained models and asking a group of data providers to participate.
We provide theoretical guarantees to show that the learned task-modelmetric distances can serve as trustworthy MRE scores, and propose synergisticlearning algorithms and models for general learning tasks.
Experiments show thatthe MRE models learned by synergistic learning can generate signiﬁcantly morereliable MRE scores than existing approaches for small-data transfer learning
Introduction
When selecting a model from a big model pool for doing model transfer, one has to do modelreusability evaluation (MRE) ﬁrst: evaluating the transfer learning performance of the models to thetarget task and identifying the best model.
The role of MRE is crucial since no matter how good thetransfer learning strategy is, incorrect MRE would still lead to the danger of negative transfer.
MREhas received growing attention in recent years [Achille et al., 2019, Tran et al., 2019, Nguyen et al.
,2020, Wu et al., 2020, Ding and Zhou, 2020, You et al., 2021].
But most existing studies focus onlarge-data MRE, under which the target dataset is sufﬁciently large.
In this work, we focus on small-data MRE, under which the target training datasets are small.
Reusing pre-trained models is essential under the small-data scenario since learning from scratchis difﬁcult.
Unfortunately, large-data MRE approaches are usually invalid for small data since theyusually focus more on simplicity and efﬁciency, but not generalization and robustness, which areessential for small-data MRE.
It is indeed challenging to obtain reliable MRE results under thesmall-data scenario due to the fundamental burden set by the laws of statistics.
Our solution is based on two observations.
First, model reusability usually depends on the taskmodel relationship other than the task-independent property of any speciﬁc small sample.
We couldgrasp this relationship by meta-learning an MRE function among tasks and models.
Second, thislearning process is often realizable in practice, in special for large model zoo platforms where sufﬁcient pre-trained models and data providers are available.
Based on these observations, we conductthe following studies in this paper:
Problem formuation.
We provide the formulation of small-data MRE and synergistic learning tolearn the MRE function by metric learning.
Synergisitc learning works for general learning scenarios beyond classiﬁcation, on which most previous MRE approaches focus.
Theoretical analysis.
We propose access risk analysis showing that the MRE model learned by syn- ergistic learning guarantees to generate reliable MRE scores.
The theory not only provides guarantee
Pre-TrainedModelSetsforTargetTasks
[Table]
ModelUniverse
[Table]
[Table]
[Figure]
the right part.
even for using non-convex deep networks as the MRE model, but also motivates feature pre-trainingin synergistic learning.
Algorithm and model design.
We propose synergistic learning algorithms and MRE model structures for general learning tasks, which have the auxiliary advantage of protecting data privacy.
Meanwhile, we propose a more elaborate MRE model strucuture for classiﬁcation.
Experimental veriﬁcation.
Experimental results show that synergistic learning can generate significantly more reliable MRE scores for small-data transfer learning than existing MRE approaches.
Problem Setup
In this section, we provide the formal deﬁnition of small-data MRE and synergistic learning.
2.1 Small-Data MRE
Denote by Z = X
× Y the observation space, in which X , Y are the input and output spacesrespectively.
A target learning task µ can be represented by a probability measure 1 µ ( x, y )
oversome Z µ ⊆ Z .
Furthermore, we assume that all µ are drawn from the task environment T , a 49 probability measure over the space of all target tasks.
In small-data MRE, target tasks µ are drawn from T .
For each µ , the objective is to learn a predictionrule f µ such that for any observation ( x, y ) ∼ µ , f µ ( x ) is close to y .
The learner of µ will be givena target training dataset S µ , a set of pre-trained models H , a model transfer strategy π and an MREfunction v π .
S µ = { z k = ( x k , y k ) }
Kk =1 is a sample drawn from µ such that the sample size K is asmall number.
H is the pre-trained model set in which each model h ∈ H is drawn from the modelenvironment M , a probability measure over the space of all pre-trained models.
We assume thatthe models in H are sampled independently from µ while different H can be given for different µ .π
( S µ , h ) is a transfer learning strategy which can be used to transfer a pre-trained model h into f µusing S µ .
The MRE function v π ( S µ , h ) is a real-valued function taking S µ and h as the inputs andoutputing the transferability score 2 .
Without loss of generality, among all possible v , we deﬁne v ∗as the optimal MRE function such that for any µ, S µ , h 1 , h 2 , v ∗ ( S µ , h 1 ) < v ∗ ( S µ , h 2 )
if and only ifh 1 has better transfer learning performance with S µ than h 2 3 .
As discussed in Section 1, we assumethat the ground-truth model transferability depends only on the target task µ but does not depend onthe random draw of S µ from µ , i.e. ∀ S µ , S " µ ∼ µ, v ∗ ( S µ ) = v ∗ ( S " µ ) .
Thus we can use the notationv ∗ ( µ, h ) instead of v ∗ ( S µ , h ) for any S µ , h .
2.2 Synergistic Learning
Synergistic learning is the preparation stage for learning an MRE function ˆ v that accurately approximate v ∗ before any MRE problem deﬁned in Section 2.1 starts.
A synergistic learning process workswith the following three prerequisites:
• A set of pre-trained models H SynLearn
= { h k } N m k
=1 drawn from M are given.
Note that themodels in this set could be different from models given in any future MRE problems; • A set of data providers D
=
{Z d } Dd =1 participate in learning, such that each data providerrepresents an observation subspace Z d ⊆ Z .
Furthermore, ∪ Z d = Z .
We deﬁne theclosure of any task µ ∼ T as ¯ Z µ =
min L {Z d 1 , Z d 2 , . . .
Z d L } s.t.
Z µ ⊆ ∪ Ll =1 Z d l .•
A validation MRE function v valid is provided for synergistic learning.
v valid is unbiasedbetween any task and model 4 : ∀ µ, h, v valid ( µ, h ) =
E S µ ∼ µ [ v valid ( S µ , h )]
= v ∗ ( µ, h ) .
The basic idea of synergistic learning is to establish a metric ˆ v ( S µ , h ) between the target trainingdatasets and the models so that the metric distance could be used as the MRE score.
In Section 4,we will show that learning ˆ v only requires the data providers to answer two kinds of queries:
• SQUERY
( ψ s , Z d , µ, M ) : a single data provider Z d is queried to sample M observationsfrom a given probability measure µ over Z d , and return the output of ψ s ( { z i } M i =1 ) , a function of the observations.• MQUERY ( ψ m , {Z d i } Qi =1 , { µ i } Qi =1 , { M i } Qi =1 ) : multiple data providers {Z d i } Qi =1 are queriedjointly.
Each data provider Z d i sample M i observations from a given probability measure
µ i over Z d i .
Aggregating the sampled observations from all data providers, the output of ψ m ( { z 1 ,j } M 1 j =1 , { z 2 ,j } M 2 j =1 , . . .
, { z Q,j } M Q j =1 ) , a function of all observations, is returned. Furthermore, ψ
m cannot be realized by aggregating multiple SQUERY .
The main difference between these two queries lies in the scope of involved data providers, which isimportant when the data privacy is a sensitive factor.
SQUERY is answered by a single data provider.
How this kind of queries can be answered in privacy-guaranteed ways receives many studies in theliterature [Koneˇcn ý et al., 2016, Zinkevich et al., 2010].
In contrast, MQUERY needs to be answeredby multiple data providers jointly.
Protecting data privacy is much harder in this situation since theirdata need to be aggregated.
For this reason, we set an auxiliary target of reducing the number oftimes to use MQUERY .
In this section, we discuss the theoretical foundation of synergistic learning. Recall that our goal is tolearn ˆ v that approximates v ∗ . We assume that ˆ v consists of three modules: task representation module g µ ( S µ ) , model representation module g µ ( h ) and metric module d θ ( c ( S µ ) , c ( h )) in which c ( S µ )and c ( h ) are outputs of g µ ( S µ ) and g h ( h ) meanwhile θ is the learnable parameter of d θ ( c ( S µ ) , c ( h )) .
The task and model representation modules transfer target training dataset S µ and pre-trained modelh into their representations c ( S µ ) and c ( h ) .
The metric module then calculates the metric distancebetween c ( S µ ) and c ( h ) as the MRE score.
In this section, we assume that the two representationmodules are ﬁxed and focus on learning the metric module.
Thus the learnable parameter for ˆ v is θ .
For metric-based synergistic learning, we assume that N m pre-trained models { h k } N m k =1 sampledfrom M , N t tasks { µ i } N t i =1 sampled from T and N S target training datasets { S µ i ,j } N S j =1 sampledfrom each task µ i are used for training.
We also assume that for any pair of h i and S µ j ,k , the groundtruth MRE score v ∗ ( µ i , h k ) are given 5 .
Deﬁne r v ∗ ( S µ , S µ " , h, h " ; ˆ v ) as I [ ∆ v ∗ ( µ, µ " , h, h " ) <0] I [ ∆ ˆ v ( S µ , S µ " , h, h " ) ≥ 0] in which ∆ v ∗ ( µ, µ " , h, h " )
= v ∗ ( µ, h ) − v ( µ " , h " ) , ∆ ˆ v ( S µ , S µ " , h, h " )
=ˆ v ( S µ , h ) − v ( S µ " , h " ) and I is the indicator function.
Since v ∗ is the ground-truth MRE function, adesirable ˆ v should minimize
[Equation]
where the expectation is taken over S µ ∼ µ, S µ " ∼ µ " , µ, µ " ∼ T , h, h " ∼ M .
While for the 111 convenience of optimization and analysis, we deﬁne our objective using the triplet surrogate loss.
Deﬁne r γ v ∗ ( S µ , S µ " , h, h " ; ˆ v ) as I [ ∆ v ∗ ( µ, µ " , h, h " ) < 0]
[ ∆ ˆ v ( S µ , S µ " , h, h " ) + γ ] + in which [ x ] + =max { x, 0 } and γ > 0 .
When γ > 1 , r γ v ∗ upper bounds r v ∗ .
Meanwhile, r γ v ∗ is consistent with r v ∗since r γ v ∗ ( S µ , S µ " , h, h " ; ˆ v ) = 0 indicates that r v ∗ ( S µ , S µ " , h, h " ; ˆ v ) = 0 .
Therefore, we deﬁne ourgoal as minimizing the following expected risk
[Equation]
A possible way to achieve this goal is to minimize the following empirical risk
[Equation]
where the summation is taken over all models, tasks and datasets used for training.
We want to know whether minimizing the empirical risk ˆ R γ (ˆ v ) indeed leads to the minimization of the expected risk
R γ (ˆ v ) .
Denote by Θ the parameter space for θ and use ˆ v θ to denote the ˆ v with learnable parameter θ .
We provide an access risk bound showing the effectiveness of minimizing the above empirical risk.
Use ¯ G θ ( S µ , S µ " , h, h " ) to denote [ ∆ ˆ v θ ( S µ , S µ " , h, h " )
+ γ ] + .
Additionally, we assume that ∇ [ ¯ G θ ( S µ , S µ " , h, h " )]
= 0 when ¯ G θ ( S µ , S µ " , h, h " ) = 0 .
Furthermore, let
[Equation]
in which the summation is taken over all tasks, datasets and models used for training and wehave ∇ − [ S µ , S µ " , h, h " ] = I [ ∆ v ∗ ( µ, µ " , h, h " ) < 0] ∇ [ ¯ G θ ( S µ , S µ " , h, h " )] .
Meanwhile, we denote
V = max i ∈ [ N t ] E S µi ∼ µ i [ c 2 ( S µ i )] .
Now we are ready to state the following theorem whose proof is 120 provided in the appendix (Section A).
Theorem 3.1.
Under Assumption A.1, there exist optimal parameter θ ∗ ∈ Θ and constant β > 0 ,∀ θ ∈ Θ , the following event
[Equation]
" holds with high probability, in which ∆ ( N t , N m , N S ) =
˜ O
(1 / √ N t , 1 / √ N m , V/ √ N S , 1 /N S ) .
Theorem 3.1 shows several interesting insights.
First, ∇ [ ˆ R ( ¯ G θ )] is the gradient norm of the metricdistance gaps from the training data.
The bound shows that the access risk will be small whenthe gradient norm tends to zero, even when the global minimum of the empirical risk has not beenreached.
This makes the bound informative when non-convex models, such as DNNs, are used.
Second, the bound has the normal O (1 / √ N t ) , O (1 / √ N m ) sample complexity dependence for bothtasks and models.
This is in agreement with our intuition such that lacking any of the tasks andmodels would lead to the failure of learning.
Finally, the most interesting take-away is the samplecomplexity for target datasets in each task, which has a dependence of the feature variance V .
If Vis small, the order becomes O (1 /N S ) instead of O (1 / √ N S ) , a signiﬁcant drop for the number ofthe training datasets.
In Section 4, we show that this result inspires us to include feature variancereduction in the synergistic learning process, which would signiﬁcantly reduce using MQUERY .
Learning Method
In this section, we introduce the synergistic learning algorithm and the MRE model.
4.1 Synergistic Learning Algorithm
The realization of synergistic learning is illustrated in Algorithm 1.
In this section, besides the stepof learning ˆ v (Line 11) discussed in Section 3, we discuss other crucial steps below in general.
Weconsider two general synergistic learning settings.
One is named isolated closure setting , whichindicates that any task closure includes a single data provider.
Otherwise, we name it groupedclosure setting , which indicates that there exist multi-data-provider task closures.
Task and model sampling (Line 8).
To generate the training data for synergistic learning, tasksand models are needed be sampled.
At this stage, no interaction to the data providers is needed.
decom d d D ; UPDATE ; $ decom 7: repeat 8:
Task
Sampling : µ 1 , µ . . , µ t ∼ T , h 1 , h h m ∼ M ; 9: Reusability Validation : g ( µ i , Z d , h k ) ← SQUERY ( Z d ) , v ∗ ( µ i , h k ) ← COMBINE [ { ψ ( µ i , Z d , h k ) }
| ¯ Z µi | d =1 ] , i ∈ [ t ] , k ∈ [ m ] , d ∈ [ ¯ Z µ i ] ; 10:
Target Dataset Generation : S µ i ← MQUERY ( ¯ Z µ i ) , i = 1 , . . .
, t ; 11: UPDATE (ˆ v ; { S µ i } ti =1 , { h k } mk =1 , { v ∗ ( µ i , h k ) } t,mi,k =1 ) ; 12: until reach end.
Reusability validation (Line 9).
The objective for reusability validation is to acquire value of v ∗between any pairs of task and model that are sampled.
In general, this can be done by sendingmodels to the data providers and estimate v ∗ using v valid .
Under the isolated closure setting, only
SQUERY is needed obviously.
We could ask the data providers to use sufﬁciently large data, makingv valid an accurate estimator.
But the key challenge appears under the grouped closure setting sincethe statistics used for calculating v valid should be obtained from multiple providers at the sametime.
To achieve this by SQUERY , we require v valid to have a special structure.
Speciﬁcally, fortask µ , let S µ ( Z d ) be a sample generated from the marginal distribution of µ over Z d , a member ofits task closure.
We require v valid ( µ, h ) =
COMBINE [ { ψ ( µ, Z d , h ) } | ¯ Z µ | d =1 ] , in which ψ ( µ, Z d , h ) =
E S µ ( Z d ) ∼ µ ( Z d ) [ ψ ( S µ ( Z d ) , h )] .
In this construction, ψ ( S µ ( Z d ) , h ) is a function over single data 156 providers and COMBINE is an aggregation function decided by speciﬁc MRE function, thus only
SQUERY is needed for calculation.
Target dataset generation (Line 10).
For isolated closure setting, this step can be done by samplingfrom a data provider using SQUERY .
However, for grouped closure setting, MQUERY is necessary forthis step since the target datasets must include the raw data.
This issue can be relieved by the insightbrought from Theorem 3.1: For each task, the number of the target data needed is closely related tothe data variance.
A feature pre-training stage can be introduced to reduce using MQUERY .
Feature pre-training (Line 2-6).
Feature pre-training is the preparation stage for synergistic learning.
It is participated by all data providers, aiming at learning a feature extractor g µ which couldoutput low-variance features for the raw data.
Once g µ is learned, it would be integrated into the
MRE model.
As a result, in synergstic learning, the data providers only need to use much fewerdata for answering MQUERY .
On the other hand, the feature pre-training stage itself should be doneby using only SQUERY .
We argue that this can be realized by using learning objectives which are decomposable w.r.t.
the data providers .
More speciﬁcally, a decomposable objective is the summationof individual objectives, such that each individual objective L ddecom only takes the data from a singledata provider Z d as its input.
Then the purpose of the queries reduces to calculating each L ddecom ,which can be done by SQUERY as expected.
Due to space limitation, we provide more details on how to implement the above synergistic learningsteps to different learning scenarios in Section B.
4.2 MRE Model
First, we introduce the basic structure of the MRE model deﬁned in Section 3, which is illustratedin (a) of Figure 2.
For simplicity, we will use shorthand notations c µ , c h to take the places ofc ( S µ ) , c ( h ) when there is no ambiguity.
In this model, the target data is ﬁrst processed by the taskfeature backbone to generate the feature c µ .
Afterwards, a task transform module is responsibleto generate the task representation φ µ which aggregates all the information from the target dataset.
For isolated closure setting, the task transform module can be simply an instance-wise averagingoperation.
In correspondence, before the model feature backbone, a model speciﬁcation generator isintroduced which can be any function that transforms a model into a vector.
The model feature c h is
[Table]
[Table]
LinearTransform
[Table]
+CkKVjjRoqv69SLDQeiz8dFNgM9SLXib+53VjE1x7CZNRbKgks6Ag5siEKCsA9ZmixPBxSjBRLP0VkSFWmJi0prkUX2SdOIsNLJNWrepcVmv355X6Td5OEY7gGE7BgSuowx00oAkEIniBV3iznq1368P6nK0WrPzmEOZgff0CdsCXWg==</latexit> w 1 ,
[Figure]
then transformed into the model representation φ h using the model transform module.
For isolatedclosure setting, the model transform module can usually be omitted, i.e. c h = φ h .
For many learning scenarios, using more elaborate MRE model structure is often useful. A typicalexample is classiﬁcation.
We provide the general formulation of small-data transfer learning forclassiﬁcation.
A target task µ has L T classes { y l } L T l =1 to distinguish, in which all the classes belongto the task class universe Y T .
Meanwhile, for a L M -way classiﬁcation model h , we denote by{ ˆ y l " } L M l " =1 the classes that the models are trained on.
We assume that all the model classes belong to 191 the model class universe
Y M , which could be different from Y T .
To reuse a pre-trained classiﬁcation 192 model on a new classiﬁcation task, We consider the popular transfer strategy of head retrain: thelearner just ﬁxes the feature backbone of the model and retrain the prediction heads using the targetdata. Head retrain is a more proper strategy than global ﬁne-tuning for small-data transfer learning.
The overall structure of the MRE model for classiﬁcation follows the basic model, but the task andmodel transform modules have more complex structures.
For the task transform module ((b) of
Figure 2), we take care of modeling the relationship among task classes.
On one hand, instead ofdirectly generating the feature representation of the whole task φ µ , the module generates the features{ φ lµ } L T l =1 for all task classes ﬁrst, and then aggregate them to form φ µ .
Any set-to-set transformmodule, such as Deep Sets [Zaheer et al., 2017] and Transformer [Vaswani et al., 2017, Ye et al.
,2020], can be used.
What is essential here is to generate the feature for one task class based on thecontext information from all other task classes.
More importantly, we propose an attention-based model transform module.
This module is motivated by our observations during studying MRE: there is a close connection between p ( y l , ˆ y l " ) andmodel reusability for classiﬁcation, which is also pointed out by recent studies [Tran et al., 2019,
Nguyen et al., 2020].
The module is illustrated in (c) of Figure 2.
First, the module takes both themodel and task class features { c l " h } L M l "
=1 , { c lµ } L T l =1 as its inputs and transform them into the modelattention-aware task class features { φ lh } L T l =1 .
Subsequently, { φ lh } L T l =1 are aggregated to form the 209 model feature φ h .
The core component of this module is the model ensembler ((d) of Figure 2).
Fora pair of model and task classes
y l , ˆ y l " , the model ensembler calculates the attention weight of y l toˆ y l " , which is w l,l " , with the inner-product attention.
And then, the model class features are linearlycombined by the attention weights to form φ lh which represents the selected model class informationon y l .
We require w l,l " to be closely related to p ( y l , ˆ y l " ) to make it represent the correlation betweeny l and ˆ y l " .
Therefore, besides the metric loss deﬁned in Equation 2, we introduce an additional attention supervision loss to supervise the learning of attention weights.
For w l,l " , we treat it as the outputprobability of a binary classiﬁer.
The training labels are generated from p ( y l , ˆ y l " ) .
To be speciﬁc,we set the label to be one if p ( y l | ˆ y l " ) > γ 1 , p (ˆ y l " | y l ) > γ 2 and zero otherwise, in which γ 1 , γ 2 aretwo thresholds.
And then, the attention supervision loss is calculated from the logistic loss.
Finally,the overall training loss is formulated as
[Equation]
in which L metric is the small-batch version of the metric loss deﬁned in Equation 2, L att is theattention supervision loss deﬁned above, and w att is the weight for L att .
[Figure] [Figure]
Experiments
In the experiments, we ﬁrst do metric visualization to verify whether synergistic learning can learnmeaningful metric space.
Furthermore, we conduct experiments for both in-dataset and cross-dataset
MRE to verify the performance of synergistic learning.
All the experiments are conducted on serverswith NVIDIA Tesla V100 GPUs.
The code is implemented using Tensorﬂow [Abadi et al., 2016]under Apache License 2.0 and is available on http://xxx.xxx.xxx.
More details of the experimentalsetups are discussed in Section C and more experimental results are included in Section D.
5.1 Metric Visualization
We adopt two ten-class datasets MNIST [LeCun et al., 1998] and CIFAR-10 [Krizhevsky, 2009]to visualize the task-model metric learned by synergistic learning.
For each dataset, we randomlygenerated 20 ﬁve-class pre-trained models for synergistic learning and another 20 models for testing.
We treat each model as ﬁve detectors for the corresponding classes and try to use synergistic learningto obtain the detector-data metric.
We consider two settings of synergistic learning.
The ﬁrst islearning with full data: all training sets are used for metric learning and there is no feature pretraining stage.
The second is learning with part data: 10% of the training set for each class isused for metric learning, and there is a feature pre-training stage involved using the full trainingset.
Figure 3 illustrates the t-SNE [Maaten and Hinton, 2008] visualizations of the learned metric,which is calculated from the testing models and instances.
We can see that meaningful metricdistance spaces are learned for both learning with full and part data: synergistic learning makes theinstances and detectors with the same classes closer.
These results provide preliminary proofs forthe effectiveness of synergistic learning.
[Table]
5.2 In-Dataset MRE
Next, we use the dSprites dataset [Matthey et al., 2017] for testing in-dataset MRE performance.dSprites consists of images generated from six latent factors.
We select three factors, shape, scale,orientation, to form the task domains.
Another two real-valued factors, position X and Y, are used asthe prediction targets.
Based on this conﬁguration, there are 720 domains and 1024 instances undereach domain in total.
For each domain, we randomly select 800 instances to form the training setwhich are used for obtaining 720 pre-trained models.
The remaining 224 instances are used as thetesting set.
We randomly select 503 domains to be the in-distribution domains and the remaining 217domains are used as the out-distribution domains.
For synergistic learning, we only use the trainingsets and models from the in-distribution domains.
v valid ( µ, h ) is set according to the mean squarederror (MSE).
We set training K = 10 , but testing K = 1 for verifying the performance under theextremely challenging situation.
During testing, we generate testing tasks using the testing sets fromeither the in-distribution domains or the out-distribution domains.
For in-distribution domains, themodels used for testing consist of only in-distribution models.
The out-distribution domains followthe similar rule.
Figure 4 shows the ranking performance comparison between using MSE directlyon the target datasets and synergistic learning (SynLearn).
It can be observed that the synergisticlearning ourperforms the naive MSE prediction signiﬁcantly: under the extreme situation wheretesting K = 1 , MSE almost fails totally in generating useful rankings, while synergistic learningcan still perform desirably.
5.3 Cross-Dataset MRE
Finally, we use CIFAR-100 [Krizhevsky, 2009] and MiniImageNet [Vinyals et al., 2016] for crossdataset MRE experiments.
Both CIFAR-100 and MiniImageNet are 100-class datasets with predeﬁned training and testing sets.
We consider two settings for experiments: Reuse CIFAR-100pre-trained models on MiniImageNet target tasks and the opposite.
For each setting, we pre-train200 20-class models on the source dataset, in which 100 models are used for synergistic learning andthe other 100 for testing.
For the target dataset, its training set is used for synergistic learning andits testing set is used for testing.
All training and testing tasks are ﬁxed to be ﬁve-way classiﬁcation.
For synergistic learning, full training set is used for metric pre-training and 10% of the trainingset is used for metric learning.
For each testing task, 50 instances are sampled from each classfor calculating the testing accuracy.
For performance evaluation, we use Kendall’s τ -coefﬁcient[Kendall, 1938] to measure the rank correlation between the MRE scores and the testing accuracy.
For emphasizing the performance on top-performed models, we also employ the weighted versionof Kendall’s τ -coefﬁcient, τ w , for which an exchange between elements with rank r and s (startingfrom zero) has weight 1 / ( r + 1) + 1 / ( s + 1)
[Vigna, 2015].
We compare the performance ofsynergistic learning (SynLearn) with three state-of-the-art MRE methods: NCE
[Tran et al., 2019],
LEEP [Nguyen et al., 2020] and LogME
[You et al., 2021].
The results are illustrated in Table 1.
It can be observed that synergistic learning signiﬁcantly outperforms other methods for small-data MRE.
Note that for results in Table 1, we ﬁx K = 10 during synergistic learning, while we observesigniﬁcantly more robust performance of SynLearn over other approaches when the testing K varies.
We conduct ablation studies to verify the effectiveness of the attention supervision, the feature pretraining, and the choice of the training K .
which are provided in Section D. We also provide moreexperimental results for cross-dataset MRE in this section.
Related Work
Direct approaches for MRE.
The direct approaches are based on the statistics calculated on thetarget training datasets as the MRE scores.
The direct approaches involve only simple statistics calculation, meanwhile no auxiliary information is used, thus are usually quite simple and efﬁcient.
Therepresentative studies of the direct approaches are [Tran et al., 2019, Nguyen et al., 2020].
In [Tranet al., 2019], the negative conditional entropy (NCE) score is proposed, which is an informationtheoretic quantity measuring the entropy for p ( y | ˆ y ) .
In [Nguyen et al., 2020], the log expectedempirical prediction (LEEP) score is proposed, which can be regarded as an improvement of NCE.
The LEEP score also closely related to p ( y | ˆ y ) .
But it uses the soft prediction probability in calculation, to take the place of the hard label assignment calculation in NCE.
Thus LEEP uses moreinformation of prediction uncertainty.
However, as shown in our experiments, both NCE and LEEPsuffer from signiﬁcant performance degeneration for small-data MRE.
This is not surprising sincethe statistics calculated from small data usually have higher variance.
Speciﬁcation-based approaches for MRE.
The speciﬁcation-based approaches use side information of the models, the model speciﬁcations , to identify model reusability [Wu et al., 2020, Dingand Zhou, 2020].
Wu et al.
[2020] proposes the RKME speciﬁcations, which are data prototypesgenerated by kernel mean embedding.
The RKME speciﬁcations reﬂect the density of the trainingdomains for the pre-trained models, which can be used to calculate the distance between source andtarget domains.
In [Ding and Zhou, 2020], the reusability indicator speciﬁcation is proposed, whichcan be used to detect in-task-domain instances.
The model speciﬁcations are designed to take auxiliary information for MRE, which can not be extracted from the model and the target data directly.
Thus the speciﬁcation-based approaches could beneﬁt from the additional side information.
However, the existing speciﬁcation-based approaches are designed for the transductive learning setting,under which the testing data of the target task are given during MRE.
This is different from theinductive learning setting studied in this paper.
Learning-based approaches for MRE.
The learning-based approaches conducts learning duringthe MRE process.
This is similar to the transductive setting considered by the speciﬁcation-basedapproaches, since the learning-based approaches need to use the testing-stage models and tasks forlearning.
In comparison, synergistic learning is only the preparation stage for testing-stage MRE.
No testing tasks and models are necessary for synergistic learning.
The representative approachesare [You et al., 2021, Achille et al., 2019].
In [You et al., 2021], the logarithm of maximum evidence(LogME) approach is proposed.
Different from NCE and LEEP which aim at ﬁnding the correlationbetween the source model predictions and the target outputs, LogME builds the correlation betweensource model features and the target outputs.
Training using target data is necessary for LogME.
Theadvantage of LogME is its wide applicability for different learning problems.
While its performancedegenerates more signiﬁcantly than NCE and LEEP under the small-data scenario.
This is likelycaused by the necessity of learning on the target data which would lead to stronger over-ﬁtting.
In [Achille et al., 2019], the Model2Vec approach is proposed.
Model2Vec uses metric learningto build task-model metric, which is similar to synergistic learning.
But Model2Vec focuses ongenerating the metric distances for a ﬁxed set of models, thus is a learning-based approach for MRE.
In comparison, synergistic learning generates the task-model metric for future MRE problems inwhich the models are unseen during synergistic learning.
Few-shot learning.
In few-shot learning (FSL) [Vinyals et al., 2016, Ha et al., 2017, Snell et al.
,2017, Finn et al., 2017], a meta-model is learned to solve future small-data learning tasks.
Usually,
FSL does not assume to use pre-trained models except for a few cases [Chowdhury et al., 2021].
For MRE, its relationship to FSL is similar to that to transfer learning: MRE can serve as a goodpreparation step for any FSL task that is allowed to use a pool of pre-trained models.
Limitations and Future Work
In this work, we proposed the synergistic learning approach for the small-data model reusabilityevaluation (MRE) problem.
This approach is based on the assumption that the model transferabilityhas little relationship with the task-independent properties of the small sample.
But in real problems,bad samples indeed exist whose task-independent properties affect transfer performance.
We believethat fundamental limits exist for dealing with these samples.
Deeper understandings of these limitsand effective ways to tackle them will be important future research topics.