Hub-Pathway: Transfer Learning from A Hub of Pre-trained Models
Anonymous Author(s)
AffiliationAddress email
Abstract
Transfer learning aims to leverage knowledge from pre-trained models to benefitthe target task.
Prior transfer learning work mainly transfers from a single model.
However, with the emergence of deep models pre-trained from different resources,model hubs consisting of diverse models with various architectures, pre-traineddatasets and learning paradigms are available.
Directly applying single-modeltransfer learning methods to each model wastes the abundant knowledge of themodel hub and suffers from high computational cost.
In this paper, we proposea Hub-Pathway framework to enable knowledge transfer from a model hub.
Theframework generates data-dependent pathway weights, based on which we assignthe pathway routes at the input level to decide which pre-trained models are activated and passed through, and then set the pathway aggregation at the outputlevel to aggregate the knowledge from different models to make predictions.
Theproposed framework can be trained end-to-end with the target task-specific loss,where it learns to explore better pathway configurations and exploit the knowledgein pre-trained models for each target datum.
We utilize a noisy pathway generatorand design an exploration loss to further explore different pathways throughoutthe model hub.
To fully exploit the knowledge in pre-trained models, each modelis further trained by specific data that activate it, which ensures its performanceand enhances knowledge transfer.
Experiment results on computer vision and reinforcement learning tasks demonstrate that the proposed Hub-Pathway frameworkachieves the state-of-the-art performance for model hub transfer learning.
Introduction
With the emergence of large-scale datasets, various large-scale deep models are developed in widerange of areas such as computer vision [18, 65, 72] and natural language processing [16, 6].
Thesemodels not only show state-of-the-art performance in the tasks that they are designed for, but alsoachieve high performance when fine-tuned on various downstream tasks with different distributions.
On the other hand, due to the requirement of large-scale annotated data, deep learning is still difficultto apply in specific domains without sufficient data [34].
Therefore, transfer learning, which aims totransfer knowledge from existing models and datasets to the target task, has attracted more and moreattention recently [75, 6].
Current deep learning paradigm tends to increase the scale of the pre-trained model to embed moreknowledge into the model, which can be transferred to more diverse tasks [25, 6].
However, infinitelyincreasing the model scale requires machines with larger storage and stronger GPUs, which areexpensive and cost much energy.
Instead, many pre-trained models from different resources arealready available, including pre-trained on diverse datasets or tasks, e.g. ImageNet for classification [55], or BERT for language modelling [16], pre-trained with diverse learning paradigms, e.g.supervised learning [29], self-supervised learning [19] or unsupervised learning [9] and with various architectures, e.g. single-stage [42] or two-stage models [28] for object detection.
These diversemodels contain abundant knowledge and introduce low cost in both training and inference.
Thus,there is a high motivation to enable knowledge transfer from all of these models.
We define theproblem as model hub transfer learning , where a model hub consists of pre-trained models fromdifferent resources.
Human beings also face the same problem of transfer learning from a model hub, where the differentpre-trained models are similar to different regions of our brain specializing for different tasks.
Whensolving a new task, human beings only call on the most relevant region of the brain to address the task.
Recently, Dean et al.
[15] also propose the outlook that the next-generation AI architecture shouldactivate different parts of the model to solve different tasks.
Motivated by these, we design pathwaysto activate different pre-trained models for prediction depending on the input data.
However, thereexist two challenges to design such pathways.
Firstly, how to explore and find the optimal pathwayconfiguration to activate the most relevant models for each target datum?
Secondly, even knowingwhich pre-trained models to use, how to aggregate and exploit the knowledge in these pre-trainedmodels?
We propose a new Hub-Pathway framework to address the above challenges, which enables effectiveknowledge transfer from a model hub.
We design an architecture with the pathway route at the inputlevel to select which pre-trained models to activate and the pathway aggregation at the output levelto aggregate the prediction from different pre-trained models, where both the pathway route andthe pathway aggregation are based on pathway weights generated data-dependently by a network.
We only activate the pathways with top k weights to avoid negative transfer of irrelevant modelsand improve the efficiency of the framework.
The framework is optimized by a task-specific lossto ensure that the pathways introduce high transfer performance.
To promote the exploration thepathways, we employ a noisy pathway generator and design an exploration loss to encourage theframework to explore and compare different pathways and find better pathway configurations.
Topromote the exploitation of the knowledge in the activated models, we further fine-tune the modelswith the specific data activating them.
In all, our contribution can be summarized as:
• We investigate the important problem of transfer learning from a model hub and propose ageneral Hub-Pathway framework which can address different situations where models maybe trained from different datasets and learning paradigms and with diverse architectures.
Wedesign data-dependent pathways to route different data to different models at the input leveland aggregate the transferred knowledge at the output level to make predictions.• We promote the exploration and exploitation of the framework to enhance more effectivetransfer learning from a model hub.
We design a noisy pathway generator and an explorationloss to encourage exploring more models and pathway configurations.
We further propose tofine-tune the activated models with specific data to fully utilize the transferred knowledge.
Related Work
Transfer Learning.
Transfer learning is a learning paradigm to improve the performance of thetarget task by transferring knowledge from source domains [49].
Early works transfer knowledgeby directly reusing features extracted by the pre-trained model [48, 17] and later works find thatfine-tuning the pre-trained model introduces higher transfer performance [2, 23].
As an easy andeffective transfer learning method, many empirical studies investigate the effectiveness of fine-tuningon various downstream tasks and scenarios, such as classification [37, 73], few-shot learning [53],medical imaging [52], object detection and instance segmentation [27].
Recently, several worksexplore the limitations of the vanilla fine-tuning method and develop new techniques to improvethe performance. To enable efficient transfer learning, residual adaptors [54] are introduced to tuneonly a few parameters for the target tasks. Parameters of fine-tuned models can be averaged, whichpromotes the performance while keeping the inference efficiency [64]. Various fine-tuning methodshave also been proposed, including regularizing the parameters [67] or features [40, 10] and exploringcategory relationships [70] or category intrinsic structures [74] by auxiliary tasks.
Several worksstudy the process of transfer learning, such as transferability of each layer in a pre-trained model [69]and the effect of intrinsic dimensions [1].
However, these studies focus on fine-tuning from a singlemodel and it remains unclear how to extend single-model transfer learning techniques to multiplepre-trained models, where simply assembling every single model is inefficient.
Transfer Learning with Multiple Models.
In the presence of multiple pre-trained models, recentworks estimate the transferability of each pre-trained model and select the best one without the needof fully fine-tuning each of them [62, 3, 47, 71, 5, 33]. But subject to the inaccurate measurementof model transferability, which is a hard problem itself, such model-level selection may degrade theperformance and lose the rich knowledge in all the models since only a single best model is utilized.
To address this problem, several works propose to aggregate the features [56, 41] or parameters [59]of all the pre-trained models.
However, all of these methods assume that the pre-trained models forma model zoo where all models have layer-to-layer correspondence or even share the same architecture,which is unrealistic for the most general case of transfer learning with arbitrary models.
We insteadpropose model hub transfer learning to remove all the constraint on the pre-trained models and realizethe most general case of transfer learning.
Conditional Computation.
Our method is also related to conditional computation or dynamic neuralnetworks [24], where parts of the network are active on a per-example basis.
A type of conditionalcomputation methods employ conditionally parameterized networks, which produce an adaptiveensemble of the model parameters [68, 11].
A series of works [35, 14, 20, 13] introduce the idea ofmultiple mixture-of-experts (MoE) [46] activated by data-adaptive gating networks.
This idea hasbeen extended to foundation models, which employ mixture-of-experts layers to form outrageouslylarge neural networks [58, 21].
Different from these methods which create mixture-of-experts fromhomogeneous networks with random initializations, our hub-pathway method simultaneously exploresthe pathways through different pre-trained models with diverse knowledge and exploits the relevantpart of knowledge to enhance transfer learning performance on the downstream tasks.
Hub-Pathway
We first define the problem of model hub transfer learning.
Then we introduce the Hub-Pathwayframework with smart designs to boost its exploration and exploitation of the model hub, simultaneously improving positive transfer of relevant knowledge and avoiding negative transfer of harmfulknowledge.
The most common transfer learning scenario considers only one single pre-trained model Θ at hand toserve the target task
D
=
{ ( x , y ) } .
In model hub transfer learning , we consider a situation where wehave a hub of pre-trained models { Θ 1 , Θ 2 , · · · , Θ m } , where all models can differ in all the aspectsincluding the pre-training datasets, pretext tasks, learning paradigms and the network architectures.
This problem setting can be generally applied in various realistic problems even if the pre-trainedmodels are obtained from quite different resources.
Motivated by the cognitive science and the pathway idea in deep networks, we propose a Hub-Pathwayframework to address the problem.
We illustrate the architecture of our framework in Figure 1.
Wemodify pre-trained models by replacing their output head layers with target-task-specific outputlayers, which form the models we will transfer from.
As our motivation stated above, different datahave different relations with the pre-trained models and should activate and transfer knowledge fromdifferent models.
To realize this idea, we employ a pathway generator network G which outputsdata-dependent pathway weights G ( x ) based on the input data x , with each dimension indicatingthe weight of one specific model in the hub.
The dense pathway weights in G ( x ) still activate all thepathways to all the pre-trained models.
To utilize the most suitable pre-trained models for the targetdata, we only keep the top k pathway weight and set the rest pathway weights as 0 :¯ G ( x ) =
f topk ( G ( x ) , k ) , where f topk ( G ( x ) , k ) i = (cid:26) G ( x ) i
G ( x ) i in the top k entries of G ( x ) otherwise (1)
Such a design has two benefits: (1) the models with useful knowledge to each target datum aretransferred but those of negative influence will be eliminated; (2) there are only data forwarding andback-propagation through the activated models, which makes both training and testing more efficient.
Based on the pathway weight G ( x ) , we assign the pathway route at the input level as I (cid:2) ¯ G ( x ) > 0 (cid:3) ,which only passes the input data through the pre-trained models with a pathway weight large than0 .
The pathways with a 0 pathway weight are blocked and the corresponding models are not usedfor the current data.
Then at the output level , an aggregator network A composes the outputs fromdifferent models with the pathway weights and outputs the final prediction of the model hub as
A (cid:0)(cid:2) ¯ G ( x ) i · Θ i ( x ) (cid:3) m i =1 (cid:1) , where the function [ · ] concatenates all the m vectors.
To train the framework, we then adopt a target-task-specific loss on the final prediction:
[Equation]
where ℓ is the target-task loss, e.g. cross-entropy loss for the classification task.
L task ensures that thederived pathway configuration and model parameters contribute to the high performance on the targettask, leading the framework to explore better pathways and exploit deeper the activated knowledge.
3.1 Enhancing Exploration on Hub Pathways
When optimized with only the task-specific loss, the pathway framework has the risk of over-fittingand falling to a local optimum by trivially activating and repeatedly updating with a few specificmodels, which ignores other potentially more useful models and wastes the rich knowledge in the hub.
Such a phenomenon is called hub collapse in this paper.
With the aim of encouraging the explorationof models with high transferability throughout the model hub, we design from the architecture leveland the loss level by employing a noisy pathway generator and introducing an exploration loss.
In order to implement a noisy pathway generator, G ( x ) is designed to embody a standard subnetwork
G p and a randomized subnetwork G n with ϵ drawn from a standard Gaussian distribution:
[Equation]
(3) where Softmax ( z ) =
exp( z ) (cid:80) z ∈
z exp( z ) and Softplus ( z )
= log(1 + exp( z )) .
The standard part G p ( x )reflects which models are preferred and the noise term G n ( x ) adds some randomness to the pathwayweights, which encourages the framework to explore other models or configurations during training.
Taking the key idea of maximum-entropy reinforcement learning that maximizes the explorationability of the agent through the maximum entropy of the policy, we define the exploration loss L exploreby directly imposing a maximum-entropy regularization on the output of the pathway generator:
[Equation]
where H ( z ) = − (cid:80) i z i · log( z i ) is the entropy function.
The loss L explore maximizes the entropy ofthe average pathway weights of all the target data.
It enforces that from the viewpoint of the wholedataset, all models have the chance to be activated and different pathways over the pre-trained modelsare fully explored if they truly promote the target performance, thereby avoiding the hub collapse.
3.2 Enhancing Exploitation in Activated Models
With the above pathway designs and losses, we can learn better data-dependent pathways for eachpiece of target data.
With the derived pathways, we know which pre-trained models to transferknowledge from, and there remains the question of how to fully exploit such knowledge.
The taskspecific loss L task utilizes and updates the knowledge in activated models, but each model contributesonly a fraction to such prediction and optimization process.
To further enhance the transfer ofknowledge and ensure the performance of the activated models, we further tune the pre-trainedmodels Θ i | m i =1 with the task-specific loss on the specific data that activate them: [Equation]
[Equation]
The above mechanism of further exploiting more useful models in the hub for each datum sharessimilar spirit with reinforcement learning in exploiting the more rewarded states in the environments.
Empowering Hub-Pathway with exploration and exploitation, the final optimization problem becomes:
[Equation]
where we use a trade-off λ between the target-task-specific loss and the exploration-encouraging losssince they have different scale.
L task and L exploit are both derived from the target-task-specific losswith the same scale and thus no trade-off is needed.
Our Hub-Pathway framework simultaneouslyexplores data-dependent pathway configurations throughout the model hub and exploits the activatedmodels by aggregating their useful knowledge and enhancing knowledge transfer to downstreamtasks.
The framework can be trained in an end-to-end way.
The computation cost at both the trainingand testing time is at most k times of the single model fine-tuning, which is much more efficient thana naive ensemble of multiple models with the computation cost linearly increasing with the hub size. [Figure]
Experiments
To verify that Hub-Pathway framework can address the general model hub transfer learning problem,we conduct experiments on tasks including image classification, facial landmark detection and reinforcement learning, which are proposed in [59].
In addition to the model hub with the homogeneousarchitecture evaluated by prior work [59], we also conduct experiments on different downstreamtasks with the model hub consisting of pre-trained models with heterogeneous architectures, whichis a more general situation that cannot be solved effectively by existing works. We have two mainhyper-parameters k and λ . We fix k = 2 and λ = 0 .
3 to avoid over-tuning of these hyper-parametersand such a setup works well on all the datasets.
The implementation code and additional experimentalresults can be found in the supplementary materials.
4.1 Transfer Learning in Image Classification
We then conduct experiments for image classification, which is a widely investigated problem withmany competitive prior work on transfer learning.
Homogeneous Model Hub.
Previous work of Zoo-Tuning [59] explores a hub of models withthe homogeneous architecture.
We follow this setting and use 5 ResNet-50 models pre-trained on5 widely-used computer vision datasets: (1) Supervised pre-trained model and (2) Unsupervisedpre-trained model with MOCO [26] on ImageNet [55], (3) Mask R-CNN [28] model for detectionand instance segmentation, (4) DeepLabV3 [8] model for semantic segmentation, and (5) Keypoint
R-CNN model for keypoint detection, pre-trained on COCO-2017 challenge datasets of each task.
Allpre-trained models are drawn from torchvision model hub [50] or from the original implementation.
Heterogeneous Model Hub.
To verify that Hub-Pathway could address more general model hubtransfer learning settings, we further conduct experiments on a model hub consisting of models withheterogeneous architectures.
We consider a model hub consisting of some other famous architecturesincluding MobileNetV3 [32], EfficientNet [61], Swin Transformer [43] and ConvNeXt [44].
For the
MobileNetV3 architecture, we use the MobileNetV3-Large-1.0 pre-trained model, which achievesthe best performance in this model family.
For the last three architectures, we use EfficientNet-B3(EffNet-B3),
Swin-Transformer-Tiny (Swin-T) and ConvNeXt-Tiny (ConvNeXt-T), which achievecomparable performance when pre-trained on ImageNet.
We also utilize the Mask R-CNN modelmentioned above to further increase the diversity of the training data and tasks in the hub. [Table]
Benchmarks.
We verify the efficacy of Hub-Pathway on 7 various classification tasks evaluatedin [59], including: (1) General classification benchmarks with CIFAR-100 [39] and COCO-70 [70];(2) Fine-grained benchmarks for aircraft ( FGVC Aircraft [45]), car ( Stanford Cars [38]) and indoorscene ( MIT-Indoors [51]) classification; (3) Specialized benchmarks collected from the DeepMind
Lab environment ( DMLab [4]) and Sentinel-2 satellite ( EuroSAT [30]).
We follow the commonfine-tuning principle and replace the last layer with a randomly initialized fully connected layer [69].
We conduct each experiment 3 times with different seeds and report the average top-1 accuracy.
Implementation Details. We follow the standard fine-tuning principle in [69]. The source taskspecific heads in the pre-trained models are replaced with newly initialized fully connected layers asthe target task-specific heads. We adopt the SGD optimizer with an initial learning rate of 0 .
01 andmomentum of 0 .
The models are trained for 15 k iterations with a batch size of 48 .
The learning rateis decayed by a rate of 0 . 1 at the 6 k-th and 12 k-th iterations. We fine-tune the pre-trained models andtrain the pathway generator and aggregator simultaneously on the target data. The pathway generatoris implemented as a 9 -layer ResNet [29]. We run the experiments 3 times and report the mean resultsof top-1 accuracy.
For all the datasets, we follow the same dataset split as in [59].
Results on Homogeneous Model Hub.
We show the results on all the benchmarks in Table 1.
We compare with fine-tuning from each single pre-trained model, the ensemble by a voting ofthe predictions of different pre-trained models after fine-tuning (Ensemble), knowledge distillationfrom the Ensemble model (Distill) and model zoo transfer learning methods including: Knowledge
Flow [41] (K-Flow), ModelSoups [64], Zoo-Tuning-L and Zoo-Tuning [59].
For our method,we activate top-2 models for each datum in all our experiments.
Hub-Pathway outperforms allthe compared methods.
In particular, different pre-trained models also show different transferperformance on different benchmarks, which demonstrates that different pre-trained models havedifferent transferability to different target tasks.
We also report a Domain Expert baseline whichchooses the best model for each task.
We observe that methods using all the pre-trained modelsoutperform Domain Expert, which demonstrates that the model hub contains much more abundantknowledge than a single model and proves the practical use of the model hub transfer learning setting.
Hub-Pathway and Zoo-Tuning both outperform Ensemble and Distill, which demonstrates that asimple ensemble of pre-trained models may not fully utilize the knowledge in the model hub and alsomay cause different pre-trained models to influence each other.
K-Flow and Zoo-Tuning-L performssimilar to Ensemble, which shows that a task-level model aggregation strategy does not maximallyutilize the knowledge in the model hub.
Finally, our Hub-Pathway outperforms Zoo-Tuning and Zoo-Tuning-L even though Zoo-Tuning is specially designed to address the situation that all thepre-trained models have the same architecture.
The observation demonstrates that Hub-Pathway findsmore transferable models and better transfer knowledge from them.
Results on Heterogeneous Model Hub.
Previous work of Knowledge Flow requires manuallydesigning the connections of different layers across different models and applying feature transformation between them.
It cannot work in such a model hub with heterogeneous architectures sincecorrespondence between layers of different pre-trained models is unknown and their features havedifferent shapes and structures.
Zoo-Tuning has an even stricter constraint of the same architecture ofthe models in the hub, which limits its application in such a more general situation.
Therefore, weonly compare our method with single-model transfer learning, Ensemble and Distill.
From Table 2, [Table] we observe that Hub-Pathway outperforms all these methods in most of the tasks.
Note that, withthe activation of 2 models for each input, the training and inference cost of Hub-Pathway is muchless than the strong Ensemble method.
The observation demonstrates that compared with existingmethods, Hub-Pathway is an effective and efficient solution to the more general situations.
4.2 Transfer Learning in Facial Landmark Detection
We conduct experiments on the Facial Landmark Detection datasets to further demonstrate that
Hub-Pathway can be applied to wide range of applications.
We use the same model hub as the imageclassification setting in Section 4.1 and transfer to three facial landmark detection tasks: 300W [57],
WFLW [66], and COFW [7], which creates a large domain gap between pre-trained models andthe target task.
We generally follow the protocol in [60] and the standard training scheme in [66].
In testing, each keypoint location is predicted by transforming the highest heat value location to theoriginal image space and adjusting it with a quarter offset in the direction from the highest responseto the second highest response [12].
Implementation Details. For the facial landmark detection experiments, we generally follow thetraining and testing protocols in [60] and the standard training scheme in [66].
The models are trainedfor 60 epochs with a batch size of 16 using the Adam optimizer.
The learning rate is set as 0 . 0001initially and is decayed by a rate of 0 . 1 at the 30 -th and 50 -th epochs. In testing, each keypointlocation is predicted by transforming the highest heat value location to the original image space andadjusting it with a quarter offset in the direction from the highest response to the second highestresponse [12].
We follow the same data split and pre-processing as in [60].
All the faces are croppedby the provided boxes according to the center location and resized to 256 × 256 . We augment thedata by ± 30 degrees in-plane rotation, 0 . 75 ∼ 1 . 25 scaling, and random flipping.
[Table]
Results.
We use the inter-ocular distance asnormalization and report the normalized meanerror (NME) for evaluation in Table 3, wheresmaller values mean better performance.
We observe that fine-tuning from a single pre-trainedmodel outperforms training from scratch, whichdemonstrates that knowledge transfer generallyimproves the target performance even thoughthe pre-trained models are trained from different tasks.
K-Flow and Zoo-Tuning performworse than Ensemble with a non-trivial margin( ≥ 0 .
in most of the tasks, which demonstrates that they fail to extract and transfer generalizable knowledge and thus do not performwell under a large domain gap between the pretrained models and the target task.
Hub-Pathwayinstead outperforms Ensemble with 0 .
1 on the WFLW task and achieves a comparable performance to Ensemble on the 300W and COFW.
Itis worth noting that Hub-Pathway is much more efficient in computation and memory than Ensemble.
The results demonstrate that Hub-Pathway could generalize across tasks even with domain gaps. [Figure]
4.3 Transfer Learning in Reinforcement Learning
[Figure] To demonstrate the generalizability of the proposed Hub-Pathway framework to various domains andtasks, we further conduct experiments on reinforcement learning models.
Benchmarks.
Current reinforcement learning algorithms have already achieved human-level performance on Atari games, but suffer from the low data efficiency with the requirement of a largeamount of interaction steps.
So we measure our method at 100k interaction steps (400k environmentsteps with action repeat of 4) on Atari, which is similar to the time to train a human learner.
We usethe Seaquest and Riverraid tasks as source tasks and construct the model hub consisting of optimalpolicies for the source tasks learned by reinforcement learning.
We transfer the model hub to 3downstream tasks: Alien, Gopher, and JamesBond.
Implementation Details.
For the learning algorithm, we follow the implementation of Data-Efficient
Rainbow [63], which modifies hyper-parameters of Rainbow [31] for data efficiency.
The model of
Data-Efficient Rainbow consists of 2 convolutional layers: 32 filters of size 5 × 5 with the strideof 5 and 64 filters of size 5 × 5 with the stride of 5 , followed by a flatten layer and 2 noisy linearlayers [22] with the hidden size of 256 .
We use Adam optimizer [36] with a learning rate of 1 × 10 − 4 .
Other hyper-parameters are kept the same as those in [63].
We repeat each experiment 5 times withdifferent seeds and report the mean and variance of the results.
Results.
We show the transfer learning performance to the three downstream tasks in Figure 2.
Wecompare with several methods including training from scratch, fine-tuning each single model, and twomulti-model transfer learning methods: Knowledge Flow [41] and Zoo-Tuning [59].
Compared with
Zoo-Tuning, which is a method tailored for this setup, Hub-Pathway slightly outperforms it on Gopherand achieves comparable performance on Alien and JamesBond.
Note that our method is generallyapplied to various architectures of pre-trained models but still works well on the homogeneousarchitecture setting.
In particular, fine-tuning from a single-model converges to the same or evenworse reward than training from scratch, which demonstrates that brutely fine-tuning does not improvethe policy performance in reinforcement learning.
Our Hub-Pathway framework fully utilizes thetransferred knowledge from the pre-trained policies and thus improves the target task performance.
4.4 Analysis
Analysis on Pathway Weights.
We visualize the average pathway weight of each pre-trained modelon all the target data in different datasets in Figure 3(a).
We observe that for different datasets, thedistribution of average pathway weight across all the pre-trained models are different.
The observationdemonstrates the challenge that different pre-trained models have different transferability to the targettask and the proposed Hub-Pathway could assign different pathway weights to different models.
The above results only demonstrate that Hub-Pathway assigns diverse weights to different pre-trainedmodels, but the quality of the pathway weights is still unknown.
To verify the quality, we fine-tuneall the pre-trained models on the target dataset and select the samples that can be classified correctlyby at least one of the models in the model hub from each dataset. Then an accurate pathway weightshould assign the highest pathway weight to at least one of the models with the correct prediction. For each sample, we make the prediction in two ways: one is to use the pre-trained model with thehighest pathway weight and the other is to use a randomly selected pre-trained model from the modelhub. We report the prediction accuracy of these two ways on the selected samples in different datasets
ImageNet MoCo MaskRCNNDeepLab Keypoint 0.
A v e r a g e
W e i g h t
CIFARCOCOAircraft (a) Average Weights
CIFAR
COCO
Aircraft
Cars 75.
A cc u r a c y ( % )
Average
Pathway (b) Quality of Weights
A cc u r a c y ( % )
COCO
Aircraft (c) Choice of k in Figure 3(b).
The results show that the selected pre-trained model with the highest pathway weightperforms much better than a uniformly sampled model, which indicates that the learned pathway canfind which pre-trained models are more transferable. Variants of Hub-Pathway. We conduct experiments on the variants of Hub-Pathway to verifythe efficacy of each component in our framework: (1) Hub-Pathway w/ random path replaces thelearned data-dependent pathway with a random pathway, which aims to demonstrate the learnedpathway actually finds the more transferable pre-trained models; (2) Hub-Pathway w/o exploreremoves the loss L explore , which may learn pathways discarding some pre-trained models forever; (3) Hub-Pathway w/o exploit removes the loss L exploit , which does not fine-tune the activated pre-trainedmodels with the exploit loss.
Hub-Pathway is the proposed framework with all the components.
Weconduct experiments under the classification setting and transfer to the task of COCO and Cars. Theresults are shown in Table 4. We observe that Hub-Pathway outperforms Hub-Pathway w/o explore, [Table]
COCO
Cars [Table] Pathway w/o explore 81. Pathway w/o exploit 80.
Hub-Pathway 84. which demonstrates that it is important toexplore more pathway configurations. HubPathway outperforms Hub-Pathway w/o exploit, which demonstrates that fine-tuningthe activated pre-trained models encouragesknowledge transfer to the target task. HubPathway outperforms Hub-Pathway w/ randompath, which demonstrates that the proposedframework learns a non-trivial pathway to select the most transferable pre-trained modelsfor the target data prediction.
Choice of k in f topk .
Our framework only activates the models within the highest top k pathwayweights.
We investigate the influence of the value k on the final target performance in Fig 3(c).
Weobserve that the performance does not change rapidly with respect to k except for k = 1 , whichmeans that aggregating the knowledge from different pre-trained models is important.
It indicatesthat we can use the Hub-Pathway framework for efficient transfer learning from the model hub.
Conclusion
In this paper, we propose a Hub-Pathway framework to enable knowledge transfer from a hub ofdiverse pre-trained models.
We design an architecture with data-dependent pathways to control whichpre-trained models are used for the current data at the input level and aggregate the outputs fromactivated pre-trained models at the output level.
The framework is optimized by a task-specific loss toensure that the pathways introduce high transfer performance.
We employ a noisy pathway generatorand design an exploration loss to encourage the framework to explore and compare different pathways.
To promote the exploitation of the knowledge in the activated models, we further fine-tune the modelswith the specific data activating them.
Experimental results on model hubs with homogeneous andheterogeneous architectures demonstrate that the proposed framework achieves the state-of-the-artperformance for model hub transfer learning.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
If you
crowdsourcing or conducted research with human subjects
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]