Pre-Trained Model Reusability Evaluation for Small-Data Transfer Learning
Abstract
We study model reusability evaluation (MRE) for source pre-trained models: evaluating their transfer learning performance to new target tasks.
In special, we focus on the setting under which the target training datasets are small, making it difﬁcult to produce reliable MRE scores using them.
Under this situation, we propose synergistic learning for building the task-model metric, which can be realized by collecting a set of pre-trained models and asking a group of data providers to participate.
We provide theoretical guarantees to show that the learned task-model metric distances can serve as trustworthy MRE scores, and propose synergistic learning algorithms and models for general learning tasks.
Experiments show that the MRE models learned by synergistic learning can generate signiﬁcantly more reliable MRE scores than existing approaches for small-data transfer learning.
Introduction
Reusing pre-trained models have played essential roles in modern learning pipelines for decreasing training cost, alleviating the requirement of big datasets, and reducing the danger of catastrophic forgetting.
The growing number of pre-trained models promotes the birth of large pre-trained model zoos making it closer towards the future learnware market [Zhou and Tan, 2022].
When selecting a model from these model zoos for doing model transfer, one has to do model reusability evaluation (MRE) ﬁrst: evaluating the transfer learning performance of the models to the target task and identifying the best model.
The role of MRE is crucial since no matter how good the transfer learning strategy is, incorrect MRE would still lead to the danger of negative transfer.
MRE has received growing attention in recent years [Achille et al., 2019, Tran et al., 2019, Nguyen et al., 2020, Wu et al., 2020, Ding and Zhou, 2020, You et al., 2021].
But most existing studies focus on large-data MRE, under which the target dataset is sufﬁciently large.
In this work, we focus on small-data MRE, under which the target training datasets are small.
Reusing pre-trained models is essential under the small-data scenario since learning from scratch is difﬁcult.
Unfortunately, large-data MRE approaches are usually invalid for small data since they usually focus more on simplicity and efﬁciency, but not generalization and robustness, which are essential for small-data MRE.
It is indeed challenging to obtain reliable MRE results under the small-data scenario due to the fundamental burden set by the laws of statistics. [Table] SynergisticLearning [Table]
[Figure]
Our solution is based on two observations.
First, model reusability usually depends on the taskmodel relationship other than the task-independent property of any speciﬁc small sample.
We could grasp this relationship by meta-learning an MRE function among tasks and models.
Second, this learning process is often realizable in practice, in special for large model zoo platforms where sufﬁcient pre-trained models and data providers are available.
Based on these observations, we conduct the following studies in this paper:
Problem formuation.
We provide the formulation of small-data MRE and synergistic learning to learn the MRE function by metric learning.
Synergisitc learning works for general learning scenarios beyond classiﬁcation, on which most previous MRE approaches focus.
Theoretical analysis.
We propose access risk analysis showing that the MRE model learned by synergistic learning guarantees to generate reliable MRE scores.
The theory not only provides guarantee even for using non-convex deep networks as the MRE model, but also motivates feature pre-training in synergistic learning.
Algorithm and model design.
We propose synergistic learning algorithms and MRE model structures for general learning tasks, which have the auxiliary advantage of protecting data privacy.
Meanwhile, we propose a more elaborate MRE model strucuture for classiﬁcation.
Experimental veriﬁcation.
Experimental results show that synergistic learning can generate significantly more reliable MRE scores for small-data transfer learning than existing MRE approaches.
Problem Setup
In this section, we provide the formal deﬁnition of small-data MRE and synergistic learning.
2.1 Small-Data MRE
Denote by Z = X
× Y the observation space, in which X , Y are the input and output spaces respectively.
A target learning task µ can be represented by a probability measure 1 µ ( x, y ) over some Z µ ⊆ Z .
Furthermore, we assume that all µ are drawn from the task environment T , a probability measure over the space of all target tasks.
In small-data MRE, target tasks µ are drawn from T .
For each µ , the objective is to learn a prediction rule f µ such that for any observation ( x, y ) ∼ µ , f µ ( x ) is close to y .
The learner of µ will be given a target training dataset S µ , a set of pre-trained models H , a model transfer strategy π and an MRE function v π .
S µ = { z k = ( x k , y k ) }
Kk =1 is a sample drawn from µ such that the sample size K is a small number.
H is the pre-trained model set in which each model h ∈ H is drawn from the model environment M , a probability measure over the space of all pre-trained models.
We assume that the models in H are sampled independently from µ while different H can be given for different µ .
π ( S µ , h ) is a transfer learning strategy which can be used to transfer a pre-trained model h into f µ using S µ .
We do not restrict the choice of π , but assume that π is ﬁxed for any target task.
The MRE function v π ( S µ , h ) is a real-valued function taking S µ and h as the inputs and outputing the transferability score.
Below we use v instead of v π to simplify notations.
Without loss of generality, among all possible v , we deﬁne v ∗ as the optimal MRE function such that for any µ, S µ , h 1 , h 2 , v ∗ ( S µ , h 1 ) < v ∗ ( S µ , h 2 )
if and only if h 1 has better transfer learning performance with S µ than h 2 . Thus we assume that the MRE score is monotonically decreasing w.r.t. transfer performance. If no special properties exist for v ∗ , estimating v ∗ would be extremely difﬁcult for small-data MRE.
Fortunately, in general, we could assume that model reusability usually depends on the task-model relationship other than the task-independent property of any speciﬁc small sample, as discussed in Section 1.
Therefore, we assume that the ground-truth model transferability depends only on the target task µ but does not depend on the random draw of S µ from µ , i.e. ∀ S µ , S " µ ∼ µ, v ∗ ( S µ ) = v ∗ ( S " µ ) .
Thus we can use the notation v ∗ ( µ, h ) instead of v ∗ ( S µ , h ) for any S µ , h .
How this assumption can be relaxed is discussed in Section 7.
2.2 Synergistic Learning
Synergistic learning is the preparation stage for learning an MRE function ˆ v that accurately approximate v ∗ before any MRE problem deﬁned in Section 2.1 starts.
A synergistic learning process works with the following three prerequisites
: • A set of pre-trained models H SynLearn
= { h k } N m k
=1 drawn from M are given.
Note that the models in this set could be different from models given in any future MRE problems; • A set of data providers D
= {Z d } Dd =1 participate in learning, such that each data provider represents an observation subspace Z d ⊆ Z .
Furthermore, ∪ Z d = Z .
We deﬁne the closure of any task µ ∼ T as
[Equation]
Ll =1 Z d l .
• A validation MRE function v valid is provided for synergistic learning.
v valid is unbiased between any task and model 2 : ∀ µ, h, v valid ( µ, h ) =
E S µ ∼ µ [ v valid ( S µ , h )]
= v ∗ ( µ, h ) .
The basic idea of synergistic learning is to establish a metric ˆ v ( S µ , h ) between the target training datasets and the models so that the metric distance could be used as the MRE score.
In Section 4, we will show that learning ˆ v only requires the data providers to answer two kinds of queries:
• SQUERY
( ψ s , Z d , µ, M ) : a single data provider Z d is queried to sample M observations from a given probability measure µ over Z d , and return the output of ψ s ( { z i } Mi =1 ) , a func- tion of the observations.
• MQUERY ( ψ m , {Z d i } Qi =1 , { µ i } Qi =1 , { M i } Qi =1 ) : multiple data providers {Z d i } Qi =1 are queried jointly.
Each data provider Z d i sample M i observations from a given probability measure µ i over Z d i .
Aggregating the sampled observations from all data providers, the output of ψ m ( { z 1 ,j } M 1 j =1 , { z 2 ,j } M 2 j =1 , . .
. , { z Q,j } M Q j =1 ) , a function of all observations, is returned.
Furthermore, ψ m cannot be realized by aggregating multiple SQUERY .
The main difference between these two queries lies in the scope of involved data providers, which is important when the data privacy is sensitive.
SQUERY is answered by a single data provider.
How this kind of queries can be answered in privacy-guaranteed ways has received many studies [Zinkevich et al., 2010, Koneˇcn ý et al., 2016].
In contrast, MQUERY needs to be answered by multiple data providers jointly.
Protecting data privacy is much harder in this situation since their data need to be aggregated.
Therefore, we set an auxiliary target of reducing the number of times to use MQUERY .
Finally, for learning ˆ v , we assume that ˆ v is formed by three parts: (1) g µ ( S µ ) , the task feature backbone; (2) g h ( h ) , the combination of the model speciﬁcation generator 3 and the model feature backbone; (3) d θ ( c µ , c h ) , the metric module, in which c µ is the output of g µ ( S µ ) , c h is the output of g h ( h ) , and θ is the learnable parameter of d θ ( c µ , c h ) .
g µ and g h transfer target training dataset S µ and pre-trained model h into their representations c µ and c h .
The metric module then calculates the metric distance between c µ and c h as the MRE score.
The details are introduced in Section 4.2.
Theoretical Analysis
In this section, we discuss the theoretical foundation of synergistic learning.
Readers who are interested more on algorithmic ideas can skip this section without affecting understanding signiﬁcantly.
In the analysis, we assume that g µ and g h are ﬁxed and focus on learning the metric module.
Thus the learnable parameter for ˆ v is θ .
For metric-based synergistic learning, we assume that N m pre-trained models { h k } N m k
=1 sampled from M , N t tasks { µ i } N t i =1 sampled from T and N S target training datasets { S µ i ,j } N S j = sampled from each task µ i are used for training.
We also assume that for any pair of h i and S µ j ,k , the ground-truth MRE score v ∗ ( µ i , h k ) are given.
Deﬁne r v ∗ ( S µ , S µ " , h, h " ; ˆ v ) as
I [ ∆ v ∗ ( µ, µ " , h, h " ) < 0] I [ ∆ ˆ v ( S µ , S µ " , h, h " ) ≥ 0] in which ∆ v ∗ ( µ, µ " , h, h " )
= v ∗ ( µ, h ) − v ∗ ( µ " , h " ) , ∆ ˆ v ( S µ , S µ " , h, h " )
=
ˆ v ( S µ , h ) − ˆ v ( S µ " , h " ) , and I is the indicator function.
Since v ∗ is the ground-truth MRE function, a desirable ˆ v should minimize
[Equation]
[Equation]
where the expectation is taken over S µ ∼ µ, S µ " ∼ µ " , µ, µ " ∼ T , h, h " ∼ M .
While for the convenience of optimization and analysis, we deﬁne our objective using the triplet surrogate loss.
Deﬁne r γ v ∗ ( S µ , S µ " , h, h " ; ˆ v ) as I [ ∆ v ∗ ( µ, µ " , h, h " ) < 0]
[ ∆ ˆ v ( S µ , S µ " , h, h " ) + γ ] + in which [ x ] + = max { x, 0 } and γ > 0 .
When γ > 1 , r γ v ∗ upper bounds r v ∗ .
Meanwhile, r γ v ∗ is consistent with r v ∗ since r γ v ∗ ( S µ , S µ " , h, h " ; ˆ v ) = 0 indicates that r v ∗ ( S µ , S µ " , h, h " ; ˆ v ) = 0 .
Therefore, we deﬁne our goal as minimizing the following expected risk
[Equation]
A possible way to achieve this goal is to minimize the following empirical risk
[Equation]
where the summation is taken over all models, tasks and datasets used for training.
We want to know whether minimizing the empirical risk ˆ R γ (ˆ v ) indeed leads to the minimization of the expected risk
R γ (ˆ v ) .
Denote by Θ the parameter space for θ and use ˆ v θ to denote the ˆ v with learnable parameter θ .
We provide an access risk bound showing the effectiveness of minimizing the above empirical risk.
Use ¯ G θ ( S µ , S µ " , h, h " ) to denote [ ∆ ˆ v θ ( S µ , S µ " , h, h " )
+ γ ] + .
Additionally, we assume that ∇ [ ¯ G θ ( S µ , S µ " , h, h " )]
= 0 when ¯ G θ ( S µ , S µ " , h, h " ) = 0 .
Furthermore, let
[Equation]
in which the summation is taken over all tasks, datasets and models used for training and we have ∇ − [ S µ , S µ " , h, h " ] = I [ ∆ v ∗ ( µ, µ " , h, h " ) < 0] ∇ [ ¯ G θ ( S µ , S µ " , h, h " )] .
Meanwhile, we denote
V = max i ∈ [ N t ] E S µi ∼ µ i [ c 2 ( S µ i )] .
Now we are ready to state the following theorem whose proof is provided in the appendix (Section A).
Theorem 3.1.
Under Assumption A.1, there exist optimal parameter θ ∗ ∈ Θ and constant β > 0 , ∀ θ ∈ Θ , the following event
[Equation]
" holds with high probability, in which ∆ ( N t , N m , N [Equation] =
[Equation]
O ( [Equation] √ [Equation]
[Equation] / √ N S , [Equation] /N S ) .
Theorem 3.1 shows several interesting insights.
First, ∇ [ ˆ R ( ¯ G θ )] is the gradient norm of the metric distance gaps from the training data.
The bound shows that the access risk will be small when the gradient norm tends to zero, even when the global minimum of the empirical risk has not been reached.
This makes the bound informative when non-convex models, such as DNNs, are used.
Second, the bound has the normal O (1 / √ N t ) , O (1 / √ N m ) sample complexity dependence for both tasks and models.
This is in agreement with our intuition such that lacking any of the tasks and models would lead to the failure of learning.
Finally, the most interesting take-away is the sample complexity for target datasets in each task, which has a dependence of the feature variance V .
If V is small, the order becomes O (1 /N S ) instead of O (1 / √ N S ) , a signiﬁcant drop for the number of the training datasets.
In Section 4, we show that this result inspires us to include feature variance reduction in the synergistic learning process, which would signiﬁcantly reduce using MQUERY .
1: Given : model set H SynLearn , data providers D .
2: if Feature Pre-Training then 3: repeat 4:
L decom ( Z d ) ← SQUERY ( Z d ) , d = 1 , . . . , D ; UPDATE ( g µ ; $ Dd =1 L decom ( Z d )) ; 5: until reach end.
6: end if 7: repeat 8:
Task & Model Sampling : µ 1 , µ 2 . .
. , µ t ∼ T , h 1 , h 2 . .
, h m ∼ M ; 9:
Reusability Validation : g ( µ i , Z d , h k ) ← SQUERY ( Z d ) , v ∗ ( µ i , h k ) ← COMBINE [ { ψ ( µ i , Z d , h k ) }
| ¯ Z µi | d =1 ] , i ∈ [ t ] , k ∈ [ m ] , d ∈ [ ¯ Z µ i ] ; 10:
Target Dataset Generation : S µ i ← MQUERY ( ¯ Z µ i ) , i = 1 , . . .
, t ; 11:
UPDATE (ˆ v ; { S µ i } ti =1 , { h k } mk =1 , { v ∗ ( µ i , h k ) } t,mi,k =1 ) to minimize Equation 2; 12: until reach end.
In this section, we introduce the synergistic learning algorithm and the MRE model.
4.1 Synergistic Learning Algorithm
The realization of synergistic learning is illustrated in Algorithm 1.
In this section, besides the metric learning step of minimizing Equation 2 (Line 11), we discuss other crucial steps below in general.
We consider two general synergistic learning settings.
One is named isolated closure setting , which indicates that any task closure includes a single data provider.
Otherwise, we name it grouped closure setting , which indicates that there exist multi-data-provider task closures.
Task and model sampling (Line 8).
To generate the training data for synergistic learning, tasks and models are needed be sampled.
At this stage, no interaction to the data providers is needed.
Reusability validation (Line 9).
The objective for reusability validation is to acquire value of v ∗ between any pairs of task and model that are sampled.
In general, this can be done by sending models to the data providers and estimate v ∗ using v valid .
Under the isolated closure setting, only
SQUERY is needed obviously.
We could ask the data providers to use sufﬁciently large data, making v valid an accurate estimator.
But the key challenge appears under the grouped closure setting since the statistics used for calculating v valid should be obtained from multiple providers at the same time.
To achieve this by SQUERY , we require v valid to have a special structure.
Speciﬁcally, for task µ , let S µ ( Z d ) be a sample generated from the marginal distribution of µ over Z d , a member of its task closure.
We require v valid ( µ, h ) =
COMBINE [ { ψ ( µ, Z d , h ) } | ¯ Z µ | d =1 ] , in which ψ ( µ, Z d , h ) =
E S µ ( Z d ) ∼ µ ( Z d ) [ ψ ( S µ ( Z d ) , h )] .
In this construction, ψ ( S µ ( Z d ) , h ) is a function over single data providers and COMBINE is an aggregation function decided by speciﬁc MRE function, thus only
SQUERY is needed for calculation.
Target dataset generation (Line 10).
For isolated closure setting, this step can be done by sampling from a data provider using SQUERY .
However, for grouped closure setting, MQUERY is necessary for this step since the target datasets must include the raw data.
This issue can be relieved by the insight brought from Theorem 3.1: For each task, the number of the target data needed is closely related to the data variance.
A feature pre-training stage can be introduced to reduce using MQUERY .
Feature pre-training (Line 2-6).
Feature pre-training is the preparation stage for synergistic learning.
It is participated by all data providers, aiming at learning a feature extractor g µ which could output low-variance features for the raw data.
Once g µ is learned, it would be integrated into the
MRE model.
As a result, in synergstic learning, the data providers only need to use much fewer data for answering MQUERY .
On the other hand, the feature pre-training stage itself should be done by using only SQUERY .
We argue that this can be realized by using learning objectives which are decomposable w.r.t.
the data providers .
More speciﬁcally, a decomposable objective is the summation of individual objectives, such that each individual objective L ddecom only takes the data from a single [Table] [Table] LinearTransform
[Table]
+CkKVjjRoqv69SLDQeiz8dFNgM9SLXib+53VjE1x7CZNRbKgks6Ag5siEKCsA9ZmixPBxSjBRLP0VkSFWmJi0prkUX2SdOIsNLJNWrepcVmv355X6Td5OEY7gGE7BgSuowx00oAkEIniBV3iznq1368P6nK0WrPzmEOZgff0CdsCXWg==</latexit> w 1 ,
[Figure] data provider Z d as its input. Then the purpose of the queries reduces to calculating each L ddecom , which can be done by SQUERY as expected.
Due to space limitation, we provide more details on how to implement the above synergistic learning steps to different learning scenarios in Section B.
4.2 MRE Model
First, we introduce the basic structure of the MRE model deﬁned in Section 3, which is illustrated in (a) of Figure 2.
In this model, the target data is ﬁrst processed by the task feature backbone to generate the feature c µ .
Afterwards, a task transform module is responsible to generate the task representation φ µ which aggregates all the information from the target dataset.
For isolated closure setting, the task transform module can be simply an instance-wise averaging operation.
In correspondence, before the model feature backbone, a model speciﬁcation generator is introduced which can be any function that transforms a model into a vector.
The model feature c h is then transformed into the model representation φ h using the model transform module.
For isolated closure setting, the model transform module can usually be omitted, i.e. c h = φ h .
For many learning scenarios, using more elaborate MRE model structure is useful, such as classiﬁcation.
Below we consider small-data transfer learning for classiﬁcation with supervised pre-trained models.
A target task µ has L T classes { y l } L T l =1 to distinguish, in which all the classes belong to the task class universe Y T .
Meanwhile, for a pre-trained L M -way classiﬁcation model h , we denote by { ˆ y l " } L M l " =1 the classes that the models are trained on.
We assume that all the model classes belong to the model class universe
Y M , which could be different from Y T .
Similar to [Tran et al., 2019,
Nguyen et al., 2020, You et al., 2021], we assume that any transfer strategy for this scenario, such as global ﬁne-tuning and head re-training, can be used.
While according to our experience, head retraining, in which the feature backbones of the pre-trained models are frozen and only the prediction heads are re-trained, is more proper than global ﬁne-tuning for small-data transfer learning.
The overall structure of the MRE model for classiﬁcation follows the basic model, but the task and model transform modules have more complex structures.
For the task transform module ((b) of
Figure 2), we take care of modeling the relationship among task classes.
On one hand, instead of directly generating the feature representation of the whole task φ µ , the module generates the features { φ lµ } L T l =1 for all task classes ﬁrst, and then aggregate them to form φ µ .
Any set-to-set transform module, such as Deep Sets [Zaheer et al., 2017] and Transformer [Vaswani et al., 2017, Ye et al., 2020], can be used.
What is essential here is to generate the feature for one task class based on the context information from all other task classes.
More importantly, we propose an attention-based model transform module.
This module is motivated by our observations during studying MRE: there is a close connection between p ( y l , ˆ y l " ) and model reusability for classiﬁcation, which is also pointed out by recent studies [Tran et al., 2019,
Nguyen et al., 2020].
The module is illustrated in (c) of Figure 2.
First, the module takes both the model and task class features { c l " h } L M l "
=1 , { c lµ } L T l =1 as its inputs and transform them into the modelattention-aware task class features { φ lh } L T l =1 .
Subsequently, { φ lh } L T l =1 are aggregated to form the model feature φ h .
The core component of this module is the model ensembler ((d) of Figure 2).
For a pair of model and task classes y l , ˆ y l " , the model ensembler calculates the attention weight of y l to ˆ y l " , which is w l,l " , with the inner-product attention.
And then, the model class features are linearly combined by the attention weights to form φ lh which represents the selected model class information on y l .
We require w l,l " to be closely related to p ( y l , ˆ y l " ) to make it represent the correlation between y l and ˆ y l " .
Therefore, besides the metric loss deﬁned in Equation 2, we introduce an additional attention supervision loss to supervise the learning of attention weights.
For w l,l " , we treat it as the output probability of a binary classiﬁer.
The training labels are generated from p ( y l , ˆ y l " ) .
To be speciﬁc, we set the label to be one if p ( y l | ˆ y l " ) > γ 1 , p (ˆ y l " | y l ) > γ 2 and zero otherwise, in which γ 1 , γ 2 are two thresholds.
And then, the attention supervision loss is calculated from the logistic loss.
Finally, the overall training loss is formulated as
[Equation]
in which L metric is the small-batch version of the metric loss deﬁned in Equation 2, L att is the attention supervision loss deﬁned above, and w att is the weight for L att .
Experiments
In the experiments, we ﬁrst do metric visualization to verify whether synergistic learning can learn meaningful metric space.
Furthermore, we conduct experiments for both in-dataset and cross-dataset
MRE to verify the performance of synergistic learning.
All experiments are conducted on servers with NVIDIA Tesla V100 GPUs.
The code 4 is implemented with TensorFlow [Abadi et al., 2016] (Apache 2.0 License).
More details of the experimental setups are discussed in Section C and more experimental results are included in Section D.
5.1 Metric Visualization
We adopt two ten-class datasets MNIST [LeCun et al., 1998] and CIFAR-10 [Krizhevsky, 2009] to visualize the task-model metric learned by synergistic learning.
For each dataset, we randomly generated 20 ﬁve-class pre-trained models, as well as 20 data providers with the same class assignments, for synergistic learning, and another 20 models for testing.
We treat each model as ﬁve detectors for the corresponding classes and try to use synergistic learning to obtain the detector-data metric.
We consider two settings of synergistic learning.
The ﬁrst is learning with full data: all training sets are used for metric learning and there is no feature pre-training stage.
The second is learning with part data: 10% of the training set for each class is used for metric learning, and there is a feature pre-training stage involved using the full training set.
Figure 3 illustrates the t-SNE [Maaten and
Hinton, 2008] visualizations of the learned metric, which is calculated from the testing models and instances.
We can see that meaningful metric distance spaces are learned for both learning with full and part data: synergistic learning makes the instances and detectors with the same classes closer.
These results provide preliminary proofs for the effectiveness of synergistic learning.
5.2 In-Dataset MRE
Next, we use the dSprites dataset [Matthey et al., 2017] for testing in-dataset MRE.
dSprites consists of images generated from six latent factors.
We select three factors, shape, scale, orientation, to form the task domains.
Another two real-valued factors, position X and Y, are used as the prediction targets.
There are 720 domains and 1024 instances under each domain in total.
For each domain, we randomly select 800 instances as the training set which are used to obtain 720 pre-trained models.
The remaining 224 instances are used as the testing set.
We randomly select 503 domains as the in-distribution domains and the remaining 217 domains as the out-distribution domains.
Under this setting, we treat each in-distribution domain as a data provider, thus there are 503 data providers in total.
For synergistic learning, we only use the training sets and models from the in-distribution domains.
v valid ( µ, h ) is set according to the mean squared error (MSE).
We set training K = 10 , but testing K = 1 for verifying the performance under the extremely challenging situation.
During testing, we generate testing tasks using the testing sets from either the in-distribution domains or the out-distribution domains.
For in-distribution domains, the models used for testing consist of only [Figure] [Figure] in-distribution models.
The out-distribution domains similar rule.
Figure 4 shows the ranking performance comparison between using MSE directly on the target datasets and synergistic learning (SynLearn).
It can be observed that the synergistic learning ourperforms the naive MSE prediction signiﬁcantly: under the extreme situation where testing K = 1 , MSE almost fails totally in generating useful rankings, while synergistic learning can still perform desirably.
5.3 Cross-Dataset MRE
Finally, we use CIFAR-100 [Krizhevsky, 2009] and MiniImageNet [Vinyals et al., 2016] for crossdataset MRE experiments.
Results on more datasets are included in Section D.
We consider two settings for experiments: Reuse CIFAR-100 pre-trained models on MiniImageNet target tasks and the opposite.
For each setting, we pre-train 200 20-class models on the source dataset, in whichmodels are used for synergistic learning and the other 100 for testing.
We use head re-training as the transfer strategy.
For the target dataset, its pre-deﬁned training set is used for synergistic learning and its pre-deﬁned testing set is used for testing.
All training and testing tasks are ﬁxed to be ﬁveway classiﬁcation.
For synergistic learning, we randomly generate 100 data providers, each of which holds the data of ﬁve target dataset classes.
Full training set is used for metric pre-training and 10% of the training set is used for metric learning.
For each testing task, 50 instances are sampled from each class to test accuracy.
For performance evaluation, we use Kendall’s τ -coefﬁcient [ken] to measure the rank correlation between the MRE scores and the testing accuracy.
For emphasizing the performance on top-performed models, we also employ the weighted version of Kendall’s τcoefﬁcient, τ w , for which an exchange between elements with rank r and s (starting from zero) has weight 1 / ( r +1)+1 / ( s +1)
[Vigna, 2015].
We compare synergistic learning (SynLearn) with three state-of-the-art MRE methods: NCE
[Tran et al., 2019], LEEP [Nguyen et al., 2020] and LogME [Table]
[You et al., 2021].
The results are illustrated in Table 1.
It can be observed that synergistic learning signiﬁcantly outperforms other methods for small-data MRE.
Note that for results in Table 1, we ﬁx K = 10 during synergistic learning, while we observe signiﬁcantly more robust performance of SynLearn over other approaches when the testing K varies.
We conduct ablation studies to verify the effectiveness of the attention supervision, the feature pre-training, and the choice of the training K .
The results are provided in Section D.
Related Work
Direct approaches for MRE.
The direct approaches are based on the statistics calculated on the target training datasets as the MRE scores.
The direct approaches involve only simple statistics calculation, meanwhile no auxiliary information is used, thus are usually quite simple and efﬁcient.
The representative studies of the direct approaches are [Tran et al., 2019, Nguyen et al., 2020].
In [Tran et al., 2019], the negative conditional entropy (NCE) score is proposed, which is an informationtheoretic quantity measuring the entropy for p ( y | ˆ y ) .
In [Nguyen et al., 2020], the log expected empirical prediction (LEEP) score is proposed, which can be regarded as an improvement of NCE.
The LEEP score also closely related to p ( y | ˆ y ) .
But it uses the soft prediction probability in calculation, to take the place of the hard label assignment calculation in NCE.
Thus LEEP uses more information of prediction uncertainty.
However, as shown in our experiments, both NCE and LEEP suffer from signiﬁcant performance degeneration for small-data MRE.
This is not surprising since the statistics calculated from small data usually have higher variance.
Learning approaches for MRE.
The learning approaches conducts learning for MRE.
Similar to the existing speciﬁcation-based approaches, the testing-stage models and tasks are used for learning.
In comparison, no testing tasks and models are necessary for synergistic learning.
The representative approaches are [You et al., 2021, Achille et al., 2019].
In [You et al., 2021], the logarithm of maximum evidence (LogME) approach is proposed.
Different from NCE and LEEP which aim at ﬁnding the correlation between the source model predictions and the target outputs, LogME builds the correlation between source model features and the target outputs.
Training using target data is necessary for LogME.
The advantage of LogME is its wide applicability for different learning problems.
While its performance degenerates more signiﬁcantly than NCE and LEEP under the small-data scenario.
This is likely caused by the necessity of learning on the target data which would lead to stronger over-ﬁtting.
In [Achille et al., 2019], the Model2Vec approach is proposed.
Model2Vec uses metric learning to build task-model metric, which is similar to synergistic learning.
But Model2Vec focuses on generating the metric distances for a ﬁxed set of models, thus is a learning approach for MRE.
In comparison, synergistic learning generates the task-model metric for future MRE problems in which the models are unseen during synergistic learning.
Few-shot learning.
In few-shot learning (FSL) [Vinyals et al., 2016, Ha et al., 2017, Snell et al., 2017, Finn et al., 2017], a meta-model is learned to solve future small-data learning tasks.
Usually,
FSL does not assume to use pre-trained models except for a few cases [Chowdhury et al., 2021].
For MRE, its relationship to FSL is similar to that to transfer learning: MRE can serve as a good preparation step for any FSL task that is allowed to use a pool of pre-trained models.
Limitations and Future Work
In this work, we proposed the synergistic learning approach for the small-data model reusability evaluation (MRE) problem.
In this section, we discuss the limitations of synergistic learning, as well as possible future research directions.
The task-independent assumption . Synergistic learning is based on the assumption that the model transferability has little relationship with the task-independent properties of the small sample.
But in real problems, bad samples indeed exist whose task-independent properties affect transfer performance.
We believe that fundamental limits exist for dealing with these samples, thus the relaxation of this assumption is challenging.
Note that Theorem 3.1 does not rely on this assumption, thus synergistic learning can be done even when it is not held.
While lacking this special property could possibly degenerate the performance.
Exploring how to tackle this challenge would be an important future research topic.
Effectiveness on more learning scenarios .
Due to the resource limitation, we only conduct experiments for supervised pre-trained models.
In recent years, reusing unsupervised pre-trained models has been increasingly popular.
MRE for unsupervised pre-trained models is a very meaningful topic, on which the research is still missing as far as we know.
Note that synergistic learning does not restrict the type of pre-trained models: it can be used whenever a ground-truth MRE function is deﬁned.
Thus it can also work for unsupervised pre-trained models in principle.
We plan to verify this point in our future researches.
Furthermore, in this paper, we only use image datasets for experiments.
Applying synergistic learning on application in other modalities is also interesting.
MRE in learnware .
Learnware [Zhou and Tan, 2022] is a growing research topic about reusing pre-trained models accommodated in a learnware market, which holds various machine learning models submitted by developers all over the world, to enable future user, who knows nothing about the models in the learnware market in advance, no need to build their own machine learning models from scratch.
One of the key ingredient of learnware is model speciﬁcation , which enable the models to be efﬁciently and adequately identiﬁed and reused, given the constraint that neither the training data of model developers nor that of users are leaked to the market.
Once a set of potentially helpful learnwares have been identiﬁed for user, there can be various ways to reuse them to help address users own task.
This paper can be viewed as providing a new way to reuse the identiﬁed learnwares.
Acknowledgements
This research is supported by NSFC (61921006, 62206245) and the Collaborative Innovation Center of Novel Software Technology and Industrialization.
We would like to thank the anonymous reviewers for their contructive suggestions, as well as Han-Jia Ye, Ming Pang and Lu Wang for helpful discussions and their thoughtful comments.