Deep Model Reassembly
Anonymous Author(s)
AffiliationAddress email
Abstract
In this paper, we explore a novel knowledge-transfer task, termed as Deep
Model Reassembly ( DeRy ), for general-purpose model reuse.
Given a collectionof heterogeneous models pre-trained from distinct sources and with diversearchitectures, the goal of DeRy , as its name implies, is to first dissect eachmodel into distinctive building blocks, and then selectively reassemble the derivedblocks to produce customized networks under both the hardware resource andperformance constraints.
Such ambitious nature of DeRy inevitably imposessignificant challenges, including, in the first place, the feasibility of its solution.
Westrive to showcase that, through a dedicated paradigm proposed in this paper, DeRycan be made not only possibly but practically efficiently.
Specifically, we conductthe partitions of all pre-trained networks jointly via a cover set optimization, andderive a number of equivalence set , within each of which the network blocks aretreated as functionally equivalent and hence interchangeable.
The equivalence setslearned in this way, in turn, enable picking and assembling blocks to customizenetworks subject to certain constraints, which is achieved via solving an integerprogram backed up with a training-free proxy to estimate the task performance.
Thereassembled models, give rise to gratifying performances with the user-specifiedconstraints satisfied.
We demonstrate that on ImageNet, the best reassemble modelachieves 78 .
6% top-1 accuracy without fine-tuning, which could be further elevatedto 83 .
2% with end-to-end training.
Our code will be made publicly available.
Introduction
The unprecedented advances of deep learning and its pervasive impact across various domains arepartially attributed to, among many other factors, the numerous pre-trained models released online.
Thanks to the generosity of our community, models of diverse architectures specializing in the sameor distinct tasks can be readily downloaded and executed in a plug-and-play manner, which, in turn,largely alleviates the model reproducing effort.
The sheer number of pre-trained models also enablesextensive knowledge transfer tasks, such as knowledge distillation in which the pre-trained modelscan be reused to produce lightweight or multi-task students.
In this paper, we explore a novel knowledge transfer task, which we coin as Deep Model
Reassembly ( DeRy ).
Unlike most prior tasks that largely focus on reusing pre-trained models asa whole, DeRy , as the name implies, goes deeper into the building blocks of pre-trained networks.
Specifically, given a collection of such pre-trained heterogeneous models or Model Zoo , DeRy attemptsto first dissect the pre-trained models into building blocks and then reassemble the building blocksto tailor models subject to users’ specifications, like the computational constraints of the derivednetwork.
As such, apart from the flexibility for model customization, DeRy is expected to aggregateknowledge from heterogeneous models without increasing computation cost, thereby preserving oreven enhancing the downstream performances.
MHSA Norm
Norm MLP
Model Zoo
Network Partition
Network Reassembly
Conv-BN-ReLU3 Conv-BN-ReLU2 Conv-BN-ReLU
Conv-BN-ReLU
Conv-BN-ReLU2 Conv-BN-ReLU
Transformer layer 3 Transformer layer 2 Transformer layer
Transformer layer
Transformerlayer
Transformer layer
Attn Norm
Norm MLP Attn Norm
Norm MLP
StitchLayer EquivalenceSet
I
EquivalenceSet II
Source Source
Source
Supervised Self-supervised
StitchLayer
Attn Norm
Norm MLP
Admittedly, the nature of DeRy per se makes it a highly challenging and ambitious task; in fact,it is even unclear whether a solution is feasible, given that no constraints are imposed over themodel architectures in the model zoo.
Besides, the reassembly process, which assumes the buildingblocks can be extracted in the first place, calls for a lightweight strategy to approximate the modelperformances without re-training, since the reassembled model, apart from the parametric constraints,is expected to behave reasonably well.
We demonstrate in this paper that, through a dedicated optimization paradigm, DeRy can be madenot only possible by highly efficient.
At the heart of our approach is a two-stage strategy that firstpartitions pre-trained networks into building blocks to form equivalence sets , and then selectivelyassemble building blocks to customize tailored models.
Each equivalence set, specifically, comprisesvarious building blocks extracted from heterogeneous pre-trained models, which are treated to befunctionally equivalent and hence interchangeable.
Moreover, the optimization of the two steps ispurposely decoupled, so that once the equivalence sets are obtained and fixed, they can readily serveas the basis for future network customization.
We show the overall workflow of the proposed DeRy in Figure 1.
It starts by dissecting pre-trainedmodels into disjoint sets of neural blocks through solving a cover set optimization problem, andderives a number of equivalence sets, within each of which the neural blocks are treated as functionallyswappable.
In the second step, DeRy searches for the optimal block-wise reassembly in a training-freemanner.
Specifically, the transfer-ability of a candidate reassembly is estimated by counting thenumber of linear regions in feature representations [43], which reduces the searching cost by 10 4times as compared to training all models exhaustively.
The reassembled networks, apart from satisfying the user-specified hard constraints, give rise to trulyencouraging results.
We demonstrate through experiments that, the reassembled model achieves >78% top-1 accuracy on Imagenet with all blocks frozen.
If we allow for finetuning, the performancescan be further elevated, sometimes even surpassing any pre-trained network in the model zoo.
Thisphenomenon showcases that DeRy is indeed able to aggregate knowledge from various models andenhance the results.
Besides, DeRy imposes no constraints on the network architectures in the modelzoo, and may therefore readily handle various backbones such as CNN, transformers, and MLP.
Our contributions are thus summarized as follows
We explore a new knowledge transfer task termed Deep Model Reassembly ( DeRy ), which enablesreassembling customized networks from a zoo of pre-trained models under certain user-specifiedconstraints.2.
We introduce a novel two-stage strategy towards solving DeRy , by first partitioning the networksinto equivalence sets and then reassembling neural blocks to customize networks.
The two stepsare modeled and solved using constrained programming, backed up with training-free performanceapproximations that significantly speed up the knowledge-transfer process.3.
The proposed approach achieves competitive performance on a series of transfer learningbenckmarks, sometimes even surpassing than any candidate in the model zoo, which, in turn, shedslight on the the universal connectivity among pre-trained neural networks.
Related Work
Transfer learning from Model Zoo.
The availability of large-scale model repositories bringsabout a new problem of transfer learning from a model zoo rather than with a single model.
Currently, there are three major solutions.
One line of works focuses on select one best model
Problem
No need to retrain
AdaptiveArchitecture No Additional Computation
Utilize All Models
HeterogeneousArchitecture
[Table]
(cid:33) (cid:37) (cid:33) (cid:37) (cid:37)
[Table]
(cid:33) (cid:37) (cid:33) (cid:37) (cid:33)
[Table]
(cid:33) (cid:37) (cid:37) (cid:33) (cid:33)
[Table]
(cid:33) (cid:37) (cid:33) (cid:33)
[Table]
(cid:37) (cid:33)
[Table]
(cid:33)
[Table]
within the zoo for deployment, either by exhaustive fine-tuning [32, 56, 65] or quantifying themodel transferability [68, 66, 45, 57, 3, 57, 4, 34] on the target task.
However, due to the unreliablemeasurement of transferability, the best model selection may be inaccurate, possibly resulting in asuboptimal solution.
The second idea was to apply ensemble methods [15, 70, 1], which inevitablyleads to prohibitive computational costs at test time.
The third approach is to adaptively fuse multiplepre-trained models into a single target model [54, 14, 55, 59].
However, those methods can onlycombine homogeneous network structures, whereas most model zoo contains a diverse range ofneural architectures.
In contrast to conventional approaches in Table 1, DeRy dissects the pre-trainedmodels into building blocks and rearranges them in order to reassemble new pre-trained models.
Neural
Representation
Similarity
Measuring similarities between
deep neural network
representations provide a practical tool to investigate the forward dynamics of deep models.
Let
X ∈ R n × d 1 and Y ∈ R n × d 2 denote two activation matrices for the same n examples.
A neuralsimilarity index s ( X, Y ) is a scalar to measure the representations similarity between X and Y ,although they do not necessarily satisfy the triangle inequality required of a proper metric.
Severalmethods including linear regression [64, 26], canonical correlation analysis (CCA) [51, 22, 50],centered kernel alignment (CKA) [33]generalized shape metrics [61].
In this study, we leverage therepresentations similarity towards function level to quantify the distance between two neural blocks.
Network Stitching.
Initially proposed by [37], model stitching aims to “plug-in” the bottom layersof one network into the top layers of another network, thus forming a stitched network [2, 12].
Itprovides an alliterative approach to investigate the representation similarity and invariance of neuralnetworks.
A recent line of work achieves competitive performance by stitching a visual transformeron top of the ResNet [56].
Instead of stitching two identical-structured networks in a bottom-topmanner, in our study, we investigate to assemble arbitrary pre-trained networks by model stitching.
Deep Model Reassembly
In this section, we dive into the proposed DeRy .
We first formulate DeRy , and then define thefunctional similarity and equivalent sets of neural blocks to partition networks by maximizing overallgroupbility.
The resulting neural blocks are then linked by solving an integer program.
3.1 Problem Formulation
Assume we have a collection of N pre-trained deep neural network models
Z = {M i } Ni =1 that eachcomposed of L i ∈ N layers of operation { F ( k ) i } L i l =1 ,
therefore M i =
F (1) i ◦ F (2) i · · · ◦ F ( L i ) i .
Eachmodel can be trained on different tasks or with varied structures.
We call Z a Model Zoo .
We define alearning task T composed of a labeled training set D tr = { x j , y j } Mj =1 and a test set D ts = { x j } Lj =1 .
Definition 1 (Deep Model Reassembly)
Given a task T , our goal is to find the best-performed L -layercompositional model M ∗ on T , subject to hard computational or parametric constraints.
We therefore formulate it as an optimization problem
[Equation]
where F ( l ) i is the l -th layer of the i -th model, P T ( M ) indicates the performance on T , and |M| ≤ Cdenotes the constraints.
For two consecutive layers with dimension mismatch, we add a singlestitching layer with 1 × 1 convolution operation to adjust the feature size.
The stitching layer structureis described in Supplementary.
No Single Wins For All.
Figure 2 provides a preliminary experiment that 8 different pre-trained
Backbone
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table]
[Figure]
[Table]
models are fine-tuned on 4 different image classification tasks.
It is clear that no single modeluniversally dominants in transfer evaluations.
It builds up our primary motivation to reassembletrained models rather than trust the “best” candidate.
Reassembly Might Win.
Table 3 compares the test performance between the reassembled modeland its predecessors.
The bottom two stages of the ResNet50 iNaturalist2021 (inat2021 sup) [58] arestitched with ResNet50 ImageNet-1k (in1k sup) stage 3&4 to form a new model for fine-tuning on
CIFAR100.
This reassembled model improves its predecessors by 0.63%/2.73% accuracy respectively.
Similar phenomenon is observed on the reassembled model between ResNet50 in1k and Swin-T in1k.
Despite its simplicity, the experiment provides concrete evidence that the neural network reassemblycould possibly lead to better model in knowledge transfer.
Reducing the Complexity.
From the overall M = (cid:80) Ni =1 L i layers, the search space of Eq 1 is ofsize L-permutations of M P ( M, L ) , which is undesirably large.
To reduce the overall search cost, weintend to partition the networks into blocks rather than the layer-wise-divided setting.
Moreover, it istime-consuming to evaluate each model on the target data through full-time fine-tuning.
Therefore,we hope to accelerate the model evaluation, even without model training.
Based on the above discussion, the essence of DeRy lies in two steps (1) Partition the networks intoblocks and (2) Reassemble the factorized neural blocks.
In the following sections, we elaborate on“what is a good partition?”
and “what is a good assembly?” .
3.2 Network Partition by Functional Equivalence
A network partition is a division of a neural network into disjoint sub-nets.
In this study, we referspecifically to the partition of neural network M i along depth into K blocks { B ( k ) i } Kk =1 so that eachblock is a stack of p layers B ( k )
i
[Equation]
and k is its stage index.
Inspired bythe hierarchical property of deep neural networks, we aim to partition the neural networks accordingto their function level, for example, dividing the network into a “low-level” block that identifiescurves and a “high-level” block that recognizes semantics.
Although we cannot strictly differentiate“low-level” from “high-level”, it is feasible to define functional equivalence .
Definition 2 (Functional Equivalence)
Given two functions B and B ′ with same input space X andoutput space Y .
d : Y × Y → R is the metric defined on Y .
For all inputs x ∈ X ,
if the outputs arethe equivalent d ( B ( x ) , B ′ ( x )) = 0 , we say B and B ′ are functional equivalent.
A function is then uniquely determined by its peers who generate the same output with the same input.
However, we can no longer define functional equivalence among neural networks, since networkblocks might have varied input-output dimensions.
It is neither possible to feed the same input tointermediate blocks with different input dimensions, nor allow for a mathematically valid definitionfor metric space [10, 5] when the output dimensions are not identical.
We therefore resort to recentmeasurements on neural representation similarity [22, 33] and define the functional similarity forneural networks.
The intuition is simple: two networks are functionally similar when they producessimilar outputs with similar inputs.
Definition 3 (Functional Similarity for Neural Networks)
Assume we have a neural similarity indexs ( · , · ) and two neural networks B : X ∈ R n × d in → Y ∈ R n × d out and B ′ :
X ′ ∈ R n × d ′ in → Y ′ ∈
R n × d ′ out .
For any two batches of inputs X ⊆ X and X ′ ⊆
X ′ with large similarity s ( X , X ′ ) >
ϵ ,the functional similarity between B and B ′ are defined as their output similarity s ( B ( X ) , B ′ ( X ′ )) .
This definition generalizes well to typical knowledge distillation (KD) [26] when d in = d ′ in , whichwe will elaborate in the Appendix.
We also show in Appendix that Def.3 provides a necessary andinsufficient condition for two identical networks.
Using the method of Lagrange multipliers, theconditional similarity in Def.3 can be further simplified to S ( B, B ′ )
=
s ( B ( X ) , B ′ ( X ′ ))
+ s ( X , X ′ ) ,which is a summation of its input-output similarity.
The full derivation is shown in the Appendix.
Finding the Equivalence Sets of Neural Blocks.
With Def.3, we are equipped with the math toolsto partition the networks into equivalent sets of blocks.
Blocks in each set are expected to have highsimilarity, which are treated to be functionally equivalent and hence interchangeable.
With a graphical notion, we represent each neural network as a path graph G ( V, E ) [20] with two 170 nodes of vertex degree 1, and the other n − 2 nodes of vertex degree 2.
The ultimate goal is to find thebest partition of each graph into K disjoint sub-graphs along the depth, that sub-graph within each 172 group has maximum internal functional similarity S ( B, B ′ ) .
In addition, we take a mild assumptionthat each sub-graph should have approximately similar size | B ( k ) i | < (1+ ϵ ) |M
i | K , where |·| indicatesthe model size and ϵ is coefficient controls size limit for each block.
We solve the above problem by 175 posing optimization
[Equation]
Partition
[Equation]
Where A ∈ N KN × K is the assignment matrix, where A ( ik,j ) =
1 denote the B ( k ) i block belongs tothe j -th equivalence set, otherwise 0.
Note that each block only belongs to one equivalence set, thuseach column sums up to 1, (cid:80) Kj =1 A ( ik,j ) = 1 .
B a j is the anchor node for the j -th equivalence set,which has the maximum summed similarly with all blocks in set j .
The inner optimization largely resembles the conventional set cover problem [27] or ( K, 1 + ϵ ) graph 181 partition problem [29] that directly partition a graph into k sets.
Although the graph partition fallsexactly in a NP-hard [23] problem, heuristic graph partitioning algorithms like Kernighan-Lin (KL)
183 algorithm
[31] and Fiduccia–Mattheyses (FM) algorithm
[18] can be applied to solve our problemefficiently.
In our implementation, we utilize a variant KL algorithm.
With a random initializednetwork partition { B ( k ) }
Kk =1 | t =0 for M at t = 0 , we iteratively find the optimal separation byswapping nodes (network layer).
Given the two consecutive block B ( k ) | t = F ( l ) i · · · ◦ F ( l + p k ) i and
B ( k +1) | t = F ( l + p k +1)
i · · · ◦ F ( l + p k + p k +1 )
i at time t , we conduct a forward and a backward neural 188 network layer swap between successive blocks, whereas the partition achieving the largest objectivevalue becomes the new partition( B ( k ) | t +1 , B ( k +1)
| t +1 ) =
arg max { J ( B ( k ) | t , B ( k +1) i | t ) , J ( B ( k ) i | f t , B ( k +1)
i | f t ) , J ( B ( k ) i | b t , B ( k +1)
i | b t ) } (6) where ( B ( k ) i | f t , B ( k +1)
i | f t )
=
B ( k ) | t F l + pk i −−−−→ B ( k +1)
i ,
[Equation] B
[Equation] B ( [Equation]
i [Equation]
For the outer optimization, we do a K-Means [40] style clustering.
With the current network partition{ B ( k ) ∗ } Kk =1 , we alternate between assigning each block to a equivalence set G j , and identifying theanchor block within each set B a j ∈ G j .
It has been proved that both KL and K-Means algorithmsconverge to a local minimum according to the initial partition and anchor selection.
We repeat theoptimization for R = 200 runs with different seeds and select the best partition as our final results.
3.3 Network Reassembly by Solving an Integer Program
As we have divided each deep network into K partitions, each belongs to one of the K equivalence 197 sets, all we want now is to find the best combination of neural blocks as a new pre-trained model undercertain computational constraints.
Consider K disjoint equivalence sets G 1 , . .
, G K of blocks to be 199 reassembled into a new deep network of parameter constraint C param and computational constraint
C FLOPs , the objective is to choose exactly one block from each group G j as well as from each network 201 stage index j such that the reassembled model achieves optimal performance on the target task without exceeding the capacity.
We introduce two the binary matrices X ( ik,j ) and Y ( ik,j ) to uniquelyidentity the reassembled model M ( X, Y ) .
X ( ik,j ) takes on value 1 if and only if B ( k ) i is chosen ingroup G j , and Y ( ik,j )
= 1 if B ( k ) i comes from the k -th block.
The selected blocks are arranged bythe block stage index.
The problem is formulated as
[Equation]
where P T is again the task performance.
Equation 10 and 11 indicates that each model only possessesa single block from each equivalence set and each stage index.
It falls exactly into a 0-1 Integer
Programming [47] problem with non-linear objective.
Conventional methods train each M ( X, Y ) toobtain P T .
Instead of training each candidate till convergence, we estimate the transfer-ability of anetwork by counting the linear regions in the network as a training-free proxy.
Estimating the Performance with Training-Free Proxy
The number of linear region [44, 21] is atheoretical-grounded tool to describe the expressivity of a neural network, which has been successfullyapplied on NAS without training [43, 7].
We, therefore, calculate the data-dependent linear region toestimate the transfer performance of each model-task combination.
The intuition is straightforward:the network can hardly learn to distinguish inputs with similar binary codes.
We apply random search to get a generation of reassembly candidates.
For a whole mini-batch ofinputs, we feed them into each network and binarilize the features vectors using a sign function.
Similar to NASWOT [43], we compute the kernel matrix K using Hamming distance d ( · , · ) and rankthe models using log(det K ) .
Since the computation of K requires nothing more than a few batchesof network forwarding, we replace P T in Equation 8 with NASWOT score for fast model evaluation.
Experiments
In this section, we first explore some basic properties of the the proposed DeRy task, and then evaluateour solution on a series of transfer learning benchmarks to verify its efficiency.
Model Zoo Setup.
We construct our model zoo by collecting pre-trained weights from Torchvision 1 ,timm 2 and OpenMMlab 3 .
We includes a series of manually designed CNN models like ResNet [25]and ResNeXt[63], as well as NAS-based architectures like RegNetY [49] and MobileNetv3 [28].
Due to recent popularity of vision transformer, we also take several well-known attention-basedarchitectures into consideration, including Vision Transformer (ViT) [16] and Swin-Transformer [38].
In addition to the differentiation of the network structure pre-trained on ImageNet, we include modelswith a variety of pre-trained strategies, including SimCLR[6], MoCov2 [8] and BYOL [19] for
ResNet50, MoCov3 [9] and MAE [24] for ViT-B. Those models are pre-trained on ImageNet1k [53],
ImageNet21K [52] and iNaturalist2021 [58], Finally we result in 21 network architectures, with28 pre-trained weights in total.
We manually identify the atomic node to satisfy our line graphassumption.
Each network is therefore a line graph composed of atomic nodes.
Implementation details.
For all experiments, we set the partition number K = 4 and the blocksize coefficient ϵ = 0 .
We sample 1/20 samples from each train set to calculate the linear CKArepresentation similarity.
The NASWOT [43] score is estimated with 5-batch average, where eachmini-batch contains 32 samples.
We set 5 levels of computational constraints, with C param ∈{ 10 , 20 , 30 , 50 , 90 } and C FLOPs ∈ { 3 , 5 , 6 , 10 , 20 } , which is denoted as DeRy ( K, C param , C FLOPs ).
Foreach setting, we randomly generated 500 candidates.
Each reassembled model is evaluate under2 protocols (1) FROZEN - TUNING .
We freeze all trained blocks and only update the parameter forthe stitching layer and the last linear classifier and (2) F ULL - TURNING .
All network parameter are updated.
All experiments are conducted on a 8 × GeForce RTX 3090 server.
To reduce the featuresimilarity calculation cost, we construct the similarity table offline on ImageNet.
The complexityanalysis and full derivation are shown in the Appendix.4.1 Exploring the Properties for Deep Reassembly Similarity, Position and Reassembly-ability.
Figure 4 validates our functional similarity,reassembled block selection, and its effect on the model performance.
For the ResNet50 trainedon ImageNet, we replace its 3 nd and 4 th stage with a target block from another pre-trainednetwork (ResNet101, ResNeXt50 and RegNetY8G), connected by a single stitching layer.
Then,the reassembled networks are re-trained on ImageNet for a 20 epochs under FROZEN - TURNINGprotocol.
The derived functional similarity in Section 3.2 is shown as the diameter of each circle.
Weobserve that , the stitching position makes a substantial difference regarding the reassembled modelperformance.
When replaced with a target block with the same stage index, the reassembled modelperforms surprisingly well, with ≥ 70% top-1 accuracy, even if its predecessors are trained withdifferent architectures, seeds, and hyperparameters.
It is also noted that, though function similarity isnot numerically proportional to the target performance, it correctly reflects the performance rankingwithin the same target network.
It suggests that our function similarity provides a reasonable criteriato identify equivalence set.
In sum, the coupling between the similarity-position-performance explainsour design to select one block from each equivalence set as well as the stage index.
We also visualizethe linear CKA [33] similarity between the R50 and the target networks in Figure 5.
An interestingfinding is that diagonal pattern for the feature similarity.
The representation at the same stage ishighly similar.
More similarity visualizations are provided in the Appendix.
Partition Results.
Due to the space limitation, the partition results of the model zoo are provided inthe Appendix.
Our observation is that, the equivalent sets tend to cluster the blocks by stage index .
For example, all bottom layers of varied pre-trained networks are within the same equivalence set.
Itprovides valuable insight that neural networks learns similar patterns at similar network stage, despiteof its architecture or training strategy.
Architecture or Pre-trained Weight.
Since our proposed DeRy searches for the architecture andweights concurrently, a natural question arises that “Do both architecture and pre-trained weights leadto the final improvement?
Or only architecture counts?”
We provide the experiments in the Appendixthat both factors contribute.
It is observed that training the DeRy architecture from scratch leads to asubstantial performance drop compared with DeRy model with both new structures and pre-trainedweights.
It validates our arguments that our reassembled models benefit from the pre-trained modelsfor efficient transfer learning.
Verifying the training-free proxy.
As the first attempt to apply the NASWOT to measure modeltransfer-ability, we verify its efficacy before applying it to DeRy task.
We adopt the score to rank10 pre-trained models on 8 image classification tasks, as well as the timm model zoo on ImageNet,shown in Figure 6.
We also compute the Kendall’s Tau correlation [30] between the fine-tunedaccuracy and the NASWOT score.
It is observed that the NASWOT score provides a reasonablepredictor for model transfer-ability with a high Kendall’s Tau correlation.
4.2 Transfer learning with Reassembled Model
Evaluation on ImageNet1k.
We first compare the reassembled network on ImageNet [53] withcurrent best-performed architectures.
We train each model for either 100 epochs as S HORT - TRAININGor a 300 epochs as FULL - TRAINING .
Except for DeRy , all models are trained from scratch.
Weoptimize each network with AdamW [39] alongside a initial learning rate of 1 e − 3 and cosinelr-decay, mini-batch of 1024 and weight decay of 0.05.
We apply RandAug [13], Mixup [69] and
CutMix [67] as data augmentation.
All model are trained [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Figure] [Figure]
[Table]
Table 7 provides the Top-1 accuracy comparison on Imagenet with various computational constraint.
We underline the best-performed model in the model zoo.
First , It is worth-noting that DeRy providevery competitive model, even under FROZEN - TURNING or S HORT - TRAINING protocol.
DeRy (4,90,20)manages to reach 78 .
6% with 1.27M parameter trainable, which provides convincing clue that theheterogeneous trained model are largely graftable.
With only S HORT - TRAINING , DeRy models alsomatch up with the full-time trained model in the zoo.
For example, DeRy (4,10,3) gets to 76 .
9%accuracy within 100 epochs’ training, surpassing all small-sized models.
The performance can befurther improved towards 78 .
4% with the standard 300-epoch training.
Second , DeRy brings aboutfaster convergence.
We compare with ResNet-50 and Swin-T under the same S HORT - TRAININGsetting in Table 9 and Figure 10.
It is clear that, by assembling the off-the-self pre-trained blocks,the DeRy models can be optimized faster than the it competitors, achieving 0.9% and 0.2% accuracyimprovement over the Swin-T model with less parameter and computational requirements.
Third ,as showcased in Figure 8, our DeRy is able to search for diverse and hybrid network structures.
DeRy (4,10,3) learns to adopt light-weight blocks like MobileNetv3, while DeRy (4,90,20) gets toa large CNN-Swin hybrid architecture.
Similar hybrid strategy has been proved to be efficient inmanual network design [42, 62].
Transfer Image classification.
We evaluate transfer learning performance on 9 natural image datasets.
These datasets covered a wide range of image classification tasks, including 3 object classificationtasks CIFAR-10 [36], CIFAR-100 [36] and Caltech-101 [17]; 5 fine-grained classification tasks
[Figure]
Flower-102 [46], Stanford Cars [35], FGVC Aircraft[41], Oxford-IIIT Pets [48] and CUB-Bird [60]and 1 texture classification task DTD [11].
We FULL - TUNE all candidate networks in the model zooand compare them with our DeRy model.
Two model selection strategies LogME [66] and LEEP [45]are also taken as our baselines.
For fair comparison, we further train the reassembled network on
ImageNet for 100 epochs to further boost the transfer performance.
Following [6, 34], we performhyperparameter tuning for each model-task combination, which are elaborated in the Appendix.
Figure 11 compares the transfer performance between our proposed DeRy and all candidate models.
By constructing models from building blocks, the DeRy generally surpasses all network trained fromscratch within the same computational constraints, even beats pre-trained ones on Cars, Aircraft,and Flower.
If allowing for pre-training on ImageNet ( DeRy +In1k), we can further promote the testaccuracy, even better than the best-performing candidate in the original model zoo (highlighted by× ).
The performance improvement rises up as parameter constraints increase, which demonstratesthe scalability of the proposed solution.
Model selection approaches like LogME and LEEP may notnecessarily get the optimal model, thus failing to release the full potential of the model zoo.
Thesefindings provide encouraging evidence that DeRy gives rise to an alternative approach to improve theperformance when transferring from a zoo of models.
[Table]
Ablation Study.
To study the influence of each stage inour solution, we conduct ablation study by replacing the(1) cover set partition and (2) training-free reassembly witha random search individually.
For the partition ablation,we randomly dissect each network into K partitions andreassemble the blocks in an order-less manner using ourtraining-free proxy.
For the reassembly ablation, weretain the cover set partition and fine-tune each randomlyreassembled network for 100 epochs.
Due to the computation limitation, we can only evaluate 25candidates for reassembly ablation.
We report the 100-epoch FROZEN - TUNING top-1 accuracy andthe search time on ImageNet in Table 2 under the DeRy (4,50,10) setting.
Note that we do not includethe similarity computation time into our account since it is computed offline
We see that the majorityof the search cost comes from the fine-tuning stage.
The training-free proxy largely alleviates thetremendous computational cost by 10 4 times, with marginal performance degradation.
On the otherhand, the cover set model partition not only improves the transfer performance but also reduces thereassembly search space from O ( (cid:81)
Ni =1 (cid:0) L i − 1 K − 1 (cid:1) ) to O (1) .
Both stages are crucial.
Conclusion
In this study, we explore a novel knowledge-transfer task called Deep Model Reassembly ( DeRy ).
DeRy seeks to deconstruct heterogeneous pre-trained neural networks into building blocks and thenreassemble them into models subject to user-defined constraints.
We provide a proof-of-conceptsolution to show that DeRy can be made not only possible but practically efficient.
Specifically, pretrained networks are partitioned jointly via a cover set optimization to form a series of equivalencesets.
The learned equivalence sets enable choosing and assembling blocks to customize networks,which is accomplished by solving integer program with a training-free task-performance proxy.
DeRy not only achieves gratifying performance on a series of transfer learning benchmarks, but alsosheds light on the functional similarity between neural networks by stitching heterogeneous models.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
[No]
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
[No]
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]