Polyhistor: Parameter-Efficient Multi-Task Adaptation for Dense Vision Tasks
Anonymous Author(s)
AffiliationAddress email
Abstract
Adapting large-scale pretrained models to various downstream tasks via fine-tuningis a standard method in machine learning.
Recently, parameter-efficient fine-tuningmethods show promise in adapting a pretrained model to different tasks whiletraining only a few parameters.
Despite their success, most existing methodsare proposed in Natural Language Processing tasks with language Transformers, and adaptation to Computer Vision tasks with Vision Transformers remainsunder-explored, especially for dense vision tasks.
Further, in multi-task settings,individually fine-tuning and storing separate models for different tasks is inefficient.
In this work, we provide an extensive multi-task parameter-efficient benchmark andexamine existing parameter-efficient fine-tuning NLP methods for vision tasks.
Ourresults on four different dense vision tasks showed that existing methods cannotbe efficiently integrated due to the hierarchical nature of the Hierarchical Vision
Transformers.
To overcome this issue, we propose Polyhistor and Polyhistor-Lite ,consisting of Decomposed HyperNetworks and Layer-wise Scaling Kernels , toshare information across different tasks with a few trainable parameters.
Thisleads to favorable performance improvements against existing parameter-efficientmethods while using fewer trainable parameters.
Specifically, Polyhistor achievescompetitive accuracy compared to the state-of-the-art while only use ∼ 10% oftheir trainable parameters.
Furthermore, our methods show larger performancegains when large networks and more pretraining data are used.
Introduction
Foundation models trained with large-scale datasets have shown the success of adapting to a varietyof downstream NLP and vision tasks [1].
As the state-of-the-art foundation models grow to billionor even trillion parameter models [2, 3, 4, 5, 6], individually fine-tuning all parameters of the modelwastes significant computational resources.
Further, for multi-task models, both fine-tuning and storing separate models for multiple tasks become infeasible on devices with low computation resources.
To alleviate this issue, several works [7, 8, 9] have proposed parameter-efficient fine-tuning methods to derive a better trade-off between trainable parameters and accuracy on downstream tasks.
By only training a small amount of parameters, these existing methods can substantially narrow the accuracy gap compared to the baseline that fine-tunes all parameters.
However, these existing approaches mainly focus on NLP tasks [10, 11, 12] or single-task adaptation on image classification [9] and their applicability to more complicated vision tasks is unclear.
On the other hand, the single-task adaptation methods [7, 10, 11, 12, 8] still need to learn and store task-wise parameters and the number of trainable parameters increase with respect to the number of tasks
Therefore, in this paper, we first conduct a thorough study on how the existing successful parameterefficient methods on NLP tasks perform on vision tasks , particularly on more challenging densevision tasks ( e.g., semantic segmentation, normals estimation).
Second, based on our findings, wethen design a novel parameter-efficient method for adaptation to multiple dense vision tasks .
Our method leverage shared modules across tasks and encourage the model to use shared informationin a more parameter-efficient manner.
To start with, we first evaluate the existing parameter-efficient methods in NLP on dense visionproblems.
We chose to apply these methods to hierarchical vision transformers (HVTs) consideringtheir state-of-the-art results on many per-pixel vision tasks [13, 14].
Through our extensive studies, wefind two limitations in these works.
First, adapter-based methods [11, 15], which have shown strongperformance on NLP parameter-efficient adaptation benchmarks, cannot be efficiently integrated with
HVTs.
This is because the parameter usage of adapters in later transformer blocks grows quadraticallywith respect to the layer scale (see Fig. 1c).
Second, the state-of-the-art multi-task parameter-efficientmethod [16] applies a hyper-network to produce the weights of adapters and shares informationacross different NLP tasks, while we find it inherently requires a large number of trainable parametersin the hyper-network (see Sec.
4.1 for further discussion).
To address the above limitations, we propose Polyhistor-Lite , which consist of two main components,
Decomposed Lite-HyperNetworks and Layer-wise Scaling Kernels .
These two methods reduce thetrainable parameters in two aspects respectively, including parameter reduction for hyper-networks inmulti-tasking architecture and parameter reduction for adapters used in HVTs.
Specifically, to reduce the parameter usage of the multi-task architecture, we decompose a hypernetwork into a pair of separate hyper-networks.
Unlike the existing approach, where a relatively largehyper-network is used to produce long vector that can be reshaped into the weight matrix of adapter,our decomposed hyper-networks individually produce two low-rank matrices that are multiplied toconstruct the adapter weights.
As a result, we can rely on this low-rank approximation to reduce theparameter usage in the hyper-network yet maintain its performance on downstream tasks.
In addition,to enable the hypernetworks shared across layers in HVTs, we factorize an adapter weight matrixinto two kernels, including Template Kernels and Scaling Kernels.
These two kernels are multipliedvia Kronecker Product to fit in with different sizes of adapters, and this is achieved by controllingthe sizes of Scaling Kernels based on the scaling of the layer/adapter (and using the same size of
Template Kernels across layers).
In this way, the parameters of adapter weights can be effectivelyreduced with a minimal sacrifice on the accuracy of downstream tasks.
To benchmark the problem, we construct a unified framework with the same implementation detailsand provide a comprehensive and fair comparison between existing parameter-efficient adaptationworks in NLP on our multi-tasking dense vision problems.
We also demonstrate that, with theintegration of our proposed Decomposed HyperNetworks and Layer-wise Scaling Kernels , we canachieve a much better trade-off between trainable parameters and accuracies compared to the existingmethods.
Specifically, most of existing methods struggled to match the performance of the simplebaseline, which individually fine-tunes the entire network for each task, while our method achievesbetter results than the simple baseline while only training less than 10% of the parameters ina model.
Compared with the state-of-the-art multi-tasking parameter-efficient adaptation method,
Hyperformer [16], our method achieves competitive performance improvement with ∼ 90% reductionin the trainable parameters of their method.
Interestingly, we also observed that our proposed methodbrings high-performance improvement when applied to the network pre-trained on the larger dataset(ImageNet-22k).
We will publicly release our code to facilitate future research.
To sum up, we list our contributions as follows:
• To the best of our knowledge, we are the first to address parameter-efficient multi-taskadaptation for vision tasks.
We develop a unified framework to benchmark several parameterefficient fine-tuning NLP methods on dense vision tasks
• We propose a novel method — Polyhistor-Lite that achieves significant performance im-
provement with very low training parameters compared to existing methods.
• We observe that our method can bring further performance improvements when applied to
models with larger pre-trained dataset or with larger backbones.
Related Works
Parameter-efficient Learning aims to adapt a pre-trained model to a new task by only training a smallnumber of parameters.
The most straightforward method is to freeze the pre-trained encoder andonly fine-tune the last layer, while, in terms of the accuracy of downstream tasks, it is still far fromfull fine-tuning.
Thus, to achieve a better trade-off between accuracy and the number of tunableparameters, several works [7, 10, 11, 12, 16, 9, 8] have proposed more parameter-efficient methods,and we summarize these works in the following paragraphs.
Single-Task Parameter-efficient Adaptation.
Several works build upon the Adapter [11], whichis a bottleneck-like module that is placed across the architecture and trained while the rest of theoriginal model is frozen.
By changing the dimension of hidden vectors, one can easily control thetrade-off between trainable parameters and accuracy.
For example, Houlsby et al.
[11] proposesto apply two adapter modules placed after the attention layers and the MLP layers respectively,while Pfeiffer et al.
[15] only use adapters after MLP layers and show more parameter-efficiency.
Furthermore, PHM-Layer [12] learns two types of matrices, one "slow" matrix shared across layersand the other "fast" matrix learned individually in different layers, to produce the adapter weight via
Kronecker Product [17].
Compacter [12] further reduces the parameters by decomposing the slowmatrix into two rank-one vectors.
Different from their goal of sharing slow matrix across layers, weapply Kronecker Product to efficiently scale up adapters to different layer scales.
In addition, there are other parameter-efficient learning works.
BitFit [7] shows simply tuning biasesin all layers improves against the linear probing.
Some other works fine-tune learnable vectors, suchas learnable vectors in input word embeddings [18] and learnable vectors integrated with keys/valuesin each layer of transformers [19].
LoRA [10] produces two low-rank matrices, which are multipliedand served as a residual of attention weight matrices.
While the above methods show favorable resultswith using fewer trainable parameters, the goal of these works is single-task adaptation.
Multi-Task Parameter-efficient Adaptation.
When multiple tasks are learned jointly, one can sharesome homogeneous information across different tasks and save the parameter usage by removingthe duplicated learned features.
To this end, Hyperformer [16] introduces a hyper-network, whichtakes as input task embeddings and produces the weights of adapters in different tasks.
Since only theparameters in the hyper-network need to be trained, the number of trainable parameters in task-wisecomponents can thus be reduced in the multi-tasking setup.
On the other hand, Sung et al.
[20] showsthat simply adding a single adapter on language transformer and sharing the adapter across tasks canachieve promising results in vision-and-language cross-modal tasks ( e.g., image captioning).
Parameter-efficient Adaptation for Vision.
Despite the promising results, most parameter-efficientlearning methods are evaluated on language transformers and NLP benchmarks, and parameterefficient learning on Vision Transformer is still an under-explored topic.
A recent work, Visual
Prompt Tuning (VPT) [9], initiates the study of parameter-efficient learning on Vision Transformers,and it follows the idea of prompt tuning in language tasks and prepends and fine-tunes some extralearnable vectors in the input space of pre-trained Vision Transformers.
VPT focuses on single-taskadaptation, while our work focuses on multi-task adaptation.
To fairly compare different parameter-efficient learning methods, He et al.
[21] present an empiricalstudy and re-evaluate parameter-efficient learning methods (BitFit, Adapters, Prefix Tuning, and
LoRA) under the same experiment configurations for NLP tasks.
Inspired by their work, we implement the aforementioned parameter-efficient NLP methods (and include more latest works [12, 16, 9])on our dense vision tasks, conduct comparative experiments, and fairly compare these methods.
Background
Hierarchical Vision Transformers.
The Vision Transformer [22] is based on transformer architectures [22] and operates on a set of patch tokens obtained from an image.
As a variant of Vision
Transformer, the Hierarchical Vision Transformer [13, 14, 23, 24, 25, 26] produces multi-scale featurerepresentations, and its hierarchical structure extracts fine-grained information and better handlesimages with scale and size variation.
These properties contribute to the promising results in severalper-pixel vision tasks, including semantic segmentation [13, 14, 26], depth estimation [27], andsaliency detection [28].
As shown in Fig.
1a, a Hierarchy Vision Transformer (HVT) consists ofseveral transformer layers, and each transformer layer is mainly composed of an attention layer and
[Figure]
an MLP layer.
Different from other transformers ( e.g., ViT [22]), a distinct characteristic of HVTs isits pyramid-like feature maps generated from different transformer blocks as shown in Fig.
1a.
Adapters.
Several parameter-efficient adaptation works [11, 15, 12, 21] build upon Adapter [11],which is a bottleneck-like module placed in transformer layers as shown in Fig.
1b.
These layers arelearnable parameters, while the rest of the model is frozen during fine-tuning.
The Adapter f a ( · )consists of a down-projection layer W down
∈ R d × n , a non-linearity function δ ( · ) , a up-projectionlayer W up ∈ R n × d , and a skip connection from the input of the h in
[Equation]
where h out ∈ R d is the output of the adapter and W =
[ W down ; W ⊺ up ] ∈ R d × 2 n represents alllearnable parameters in the adapter.
Method
Problem Setting.
Given a Hierarchical Vision Transformer pre-trained on large-scale image datasets( e.g., ImageNet [29]), our goal is to train a small number of parameters and adapt the model to themulti-tasking setting, where training data of N tasks are obtained during the training stage.
Followingthe existing works in NLP, the criteria of parameter-efficient multi-tasking learning includes theaccuracy of downstream tasks and the numbers of training parameters.
Method Overview.
We aim to improve the parameter efficiency in two aspects: (1) to efficientlyshare homogeneous information across tasks via lightweight hyper-networks (Section 4.1) and(2) to efficiently scale up adapter weights in different transformer blocks of Hierarchical Vision
Transformers (Section 4.2).
These two components are combined to improve the trade-off betweenaccuracy and training parameters in multi-tasking per-pixel vision tasks (Section 4.3).
4.1 Polyhistor : Decomposed Lightweight Hyper-networks for Multi-task Adaptation
With the goal of jointly adapting multiple NLP tasks in a parameter-efficient manner, a prior work,
Hyperformer [16], builds upon a group of adapters in different tasks and extracts task-sharinginformation via a hyper-network shared across different tasks.
Specifically, a group of task and layerwise adapters with weight parameters { W tl | t = 1 , ..., N ; l = 1 , ..., L } are individually inserted intoeach layer l of the model with L layers for all N tasks.
Then, instead of individually learning theweights of these adapters via backpropagation, Hyperformer constructs a layer-wise hyper-networkˆ W l , which takes as input a learnable task embedding V t and produce the weights of adapters W tl .
[Equation]
where Π( · ) maps a vector with size of 2 dn to a matrix with size of d × 2 n .
While Hyperformer has shown promising results on multi-task NLP benchmarks, its effectiveness onvision tasks is unclear.
In addition, since the hyper-network produces the vectorization of an adapterweight matrix (
i.e., W tl ∈ R d × 2 n ), the output dimension of the hyper-network is the order of O ( dn )and the size of the hyper-network becomes the order of O ( dnk ) , where d and n are dimensions ofthe input and bottleneck vectors and k is the size of task embeddings.
For dense vision tasks, thesize of input vectors is usually large ( e.g., 1024 in SwinTransformer-Base).
When the bottleneckdimension is set to be proportional to the input dimension ( e.g., n = dα , where α > 1 is a constant),the size of hyper-network is then quadratically increased with respect to input vectors O ( kd 2 ) .
To alleviate this issue, we propose to decompose a single hyper-network ˆ W l into a pair of lightweighthyper-networks { ˆ W pl , ˆ W ql } , each of which only produces a low-rank matrix.
We then multiply thematrices to obtain the adapter weight as shown in the top of Fig.
2a.
[Equation]
where Π p ( · ) and Π q ( · ) are matrix reshape functions and r is a matrix rank which is hyper-parametertuned according the computational budget.
Note that the matrix rank r is usually much smaller thandimensions of an adapter n or d ( i.e., r << n < d ).
In this way, with the low-rank decomposition and approximation, a heavy hyper-network ˆ W l ∈
R k × 2 dn can be reduced to two lightweight hyper-networks { ˆ W pl ∈ R k × dr , ˆ W ql ∈ R k × 2 nr } , andthe number of trainable parameters can be reduced from 2 kdn to k ( d + 2 n ) .
The number of trainableparameters in hyper-networks is reduced from quadratic to linear increasing with respect to the inputsize ( i.e., O ( kd 2 ) → O ( kd ) ).
We will discuss how this translate into practical usage and demonstratethat our method can significantly save the number of training parameters while achieving competitiveperformance compared to the vanilla Hyperformer in Sec. 5.
4.2 Layer-Wise Scaling Kernels for Hierarchical Vision Transformers
Although the number of learnable parameters is reduced by sharing two lightweight hyper-networksacross tasks, each layer of transformer still requires a pair of layer-wise hyper-networks.
To preventthe number of parameters from growing linearly with respect to the number of layers in a transformer,one can share a hyper-network across not only different tasks but also different layers (similar tothe recently introduced Hyperformer++ [16] in NLP).
However, since adapters in different blocks of
Hierarchical Vision Transformers have different dimensions, such a property restricts us from usingthe same pair of hyper-networks to produce the weights of adapters in different transformer layers.
To overcome this issue, we introduce Layer-wise Scaling Kernels to enable sharing a hyper-networkacross layers.
To be more specific, as shown in Fig.
1, Hierarchical Vision Transformers have fourtransformers blocks, and each transformers block (indexed by b ) has multiple transformer layers witha block-wise scale s b = 2 b − 1 and a output size ( H 4 s b , W 4 s b , s b C ) .
However, the increasing channelsizes in transformer blocks cause two problems.
First, as shown in Fig.
1, the sizes of adapter weightsincrease quadratically with respect to the channel size/input size.
As a result, the adapters in the laterblocks would require much more trainable parameters than the earlier blocks.
Second, as mentionedabove, due to the different sizes of adapters in different transformer layers, a single hyper-networkcannot produce multiple adapters in different layers and thus cannot be directly shared across layers.
As shown in the bottom of Figure 2a, to address these two problems, we propose to factorize theweight matrix of a adapter W l of the layer l into a set of Template Kernels ˜ W il and Scaling Kernelsκ il .
To efficiently scale up adapter weights in different transformer layers, we resort to Hypercomplex
[Figure]
applies
HyperNetworks reduce the number of training parameters Section multiplication [17] and use Kronecker Product to integrate these types of matrices.
[Equation]
where s ψ ( l ) and ψ ( l ) are the scale and the index of transformer block where transformer layer l islocated, and ⊗ is Kronecker Product matrix operation.
The sizes of Template Kernels are the sameacross layers, while the sizes of Scaling Kernels depend on the block-wise scale s ψ ( l ) .
In other words, the purpose of Scaling Kernels is to scale up the Template Kernels and fit them intothe adapters in layers with different scales.
This decomposition not only reduces the parameter usagein a single adapter but also shows the potential to reduce parameters with a shared hyper-network.
4.3 Polyhistor-Lite : Lightweight Hypernetworks for Hierarchical Vision Transformers
We have described how to reduce the training parameters for multi-tasking adaptation in Section 4.1and Hierarchical Vision Transformers in Section 4.2.
To perform parameter-efficient adaptation formulti-tasking per-pixel vision tasks, we integrate these two components to obtain our final framework.
Specifically, as shown in Fig.
2b, we use a single pair of hyper-networks { ˆ W p , ˆ W q } shared acrossdifferent layers and different tasks.
For the task t , the hyper-networks take as input a set of trainablelayer embeddings { ˜ V il } s ψ ( l ) i =1 and a task embedding ˜ V t and produces two low-rank matrices, whichare multiplied and derive a set of Template Kernels of adapters in different layers and tasks.
[Equation]
To derive the parameters of an adapter in each layer, we learn another set of Scaling Kernels andcombine them with the Template Kernels via Kronecker Product.
[Equation]
With the integration of Lightweight HyperNetworks and Layer-wise Scaling Kernels, our frameworkcan efficiently reduce the trainable parameters in multi-task adaptation for dense vision tasks.
Weprovide two variants of our method.
Polyhistor solely uses the Decomposed HyperNetworks, and
Polyhistor-Lite combines both Decomposed HyperNetworks and Layer-Wise Scaling Kernels.
Experiments
5.1 Implementation Details
Dataset.
We follow prior works [30, 31] on multi-task learning for dense prediction tasks and consider
PASCAL-Context [32] to construct our multi-task efficient adaptation for per-pixel benchmark.
Weevaluate all methods on four per-pixel tasks, 21 -class semantic segmentation, 7 -class human partsegmentation, surface normals estimation, and saliency detection.
Our evaluation metrics includethe mean intersection-over-union (mIoU) for semantic segmentation, human part segmentation, andsaliency detection and the mean error (mErr) for surface normals estimation.
Model Architecture.
For the encoder, we use Swin-Transformer [13] due to its strong performancein different vision tasks and the popularity in the vision community.
Our decoders for different densetasks are based on the All-MLP decoder of Segformer [14], which uses simple linear layers andbilinear upsampling layer to efficiently perform dense vision tasks, and we adapt the number ofoutput dimension to different tasks.
Training.
To train our model, we use the commonly-used losses for each task.
Specifically, we usethe standard per-pixel cross-entropy for semantic segmentation and human part segmentation, L1loss for surface normals estimation, and balanced cross-entropy for saliency detection.
For a faircomparison, we experiment on a unified codebase implementation with the same loss functions andtraining iterations for all baselines and our method.
5.2 Baselines
Single-task full fine-tuning uses an individual pretrained model for each task, and Fine-tuningdecoders freezes the feature backbone and only fine-tunes task-wise decoders for different tasks.
For single-task adaptation methods (Bitfit, VPT, PHM-Layer, Compacter, Compacter++, Adapter,
Low-rank Adapter, and LoRA)
, we place task-wise modules for each task.
Bitfit [7] tune the biases in all layers, and, specifically for Swin-Transformer, and we also tune biasesin patch merging layers and patch projection layers.
VPT [9] inserts tunable embeddings in the first input layer (VPT-shallow) and all layers (VPT-deep),and we select the best hyper-parameter ( i.e., 50 embeddings per layer) for all results.
PHM layer [12] shares a slow matrix for all layers and learns a fast matrix for each layer and placemodules after attention and MLP layers, Compacter [12] further decomposes the fast matrix intotwo low-rank vectors, and Compacter++ [12] only places modules after MLP layers.
LoRA [10] applied the low-rank decomposition on attention layers, and we select rank r = 4 and theadapter output scale ( i.e., 4), which performs the best.
Adapter [11, 21] placed task-wise bottleneck-like modules into transformer layers, and SharedAdapter [20] share an adapter across different tasks.
Hyperformer [16] applied a hyper-network and produce the weights for adapter, and we present theresults with different adapter bottleneck dimensions.
Since adapters in different layers have differentdimensions, Hyperfomer++ cannot be simply adapted to Hierarchical Vision Transformers.
[Figure]
5.3 Experiment results on Multi-Task Adaptation
We evaluate all methods by computing relative improvements against Single-Task Full Fine-tuningand averaging across four tasks.
Since all methods use and train all parameters of task-wise decoders,we provide two values of trainable parameters, one for the encoder and the other for the whole model.
As presented in Fig.
3 and Table 1, among all experimental methods, Hyperformer performs thebest and achieves +2 .
64% on average for the four downstream tasks, but it requires 72 M trainableparameters in the encoder.
On the other hand, our Polyhistor achieves competitive results ( +2 .
34% ),while we only need 6 .
41 M trainable parameters in the encoder, which is less than one-tenth of the
Hyperformer.
Our Polyhistor-Lite can further reduce the trainable parameters to 0 .
41 M by integratingthe Layer-wise Scaling Kernels and sharing the hypernetwork across layers, and it achieves +1 .
74%and is higher than all other methods using a similar amount of trainable parameters ( e.g., BitFit, VPT,
Shared Adapter, PHM layer, Compacter, LoRA, and Low-rank Adapter).
We also found that, while the prior parameter-efficient adaptation vision method, VPT, presentedpromising results on single-task image classification [9], it does not show significant improvementsagainst the baseline that only fine-tuning decoders in our multi-task dense vision benchmark.
Apotential reason is that, compared to the image classification benchmarks which focus more on thesame task with input shifts, our benchmark focuses more on different task outputs.
This makes VPT,which adds the learnable parameters in the input space, unable to address the difference in the outputspace and adapt to the new tasks.
While Hyperformer achieves the best performance improvement against Single-Task Full Fine-tuning,it requires a larger number of parameters than other methods.
This leads to a natural question: Doesreducing the parameters in Hyperformer derive a better trade-off between the number of tunableparameters and performance improvements?
Therefore, we decrease its size and examine whether itcan maintain its performance with fewer trainable parameters.
We also did a similar experiment onthe other existing methods, ( e.g., Adapter, Shared Adapter, Compacter, and Compacter++), in whichwe increase the number of tuning parameters by increasing the dimension of the hidden vector in theadapter modules (increasing the down-projection ratio).
As shown in Fig.
3b, we find that simplytuning the hyper-parameters in baseline methods cannot obtain a better tradeoff between the numberof trainable parameters and performance improvement.
For example, when the number of tuningparameters in the encoder of Hyperformer is reduced to 20 .
15 M, its averaged relative improvementshrinks to only − 0 .
These experiments show that our methods achieve a better performanceimprovement with a fewer number of parameters than the baseline methods.
5.4 Ablation studies and analyses
Different feature backbones.
To verify that our proposed method is also applicable to other largersize model architecture, we also experiment SwinTransformer-Base.
For a fair comparison, we use thesame pretraining dataset, ImageNet-1k [29], on both SwinTransformer-Tiny and SwinTransformerBase.
As shown in Fig.
4a, we find that our method achieves a larger improvement against Single-task
[Table]
-2.71 % Low-rank adapter 0.34 / 2.
Shared Adapter [20] 2.20 / 4.
-1.83 % Hyperformer [16] 72.77 / 75.
Polyhistor (Ours) 6.41 / 8.
2.34 % Polyhistor-Lite (Ours) 0.41 / 2.
[Figure]
Full Fine-tuning when a larger feature backbone is used, and this shows a potential of obtaining moreimprovements when applying our method to a larger model architecture.
Different Pretraining data.
We also examine how our proposed method performs when differentpretraining data are used.
Specifically, we apply our method to SwinTransformer-Base pretrainedwith ImageNet-1k and ImageNet-22k.
As demonstrated in Fig.
4b, we find that our method obtains alarger performance improvement when more pretraining data was used, and this shows a potential ofderiving more improvements by using more pretraining data.
We present more ablation studies, analyses, and implementation details in Appendix.
Conclusion
We proposed Polyhistor and Polyhistor-Lite — parameter-efficient fine-tuning methods for densevision tasks.
We showed that most of the existing parameter-efficient single-task adaptation methodsachieved lower performance compared with Single-task Full Fine-tuning, and the state-of-the-artmulti-task adaptation method achieve favorable results while using a large number of tunable parameters.
Compared to these existing approaches, our proposed methods do not only achieve a competitiveperformance gain to the state-of-the-art but also only use a very limited amount of tunable parameters.
The potential limitation of our method is searching for suitable hyper-parameters, which is a commonlimitation among all parameter-efficient learning methods.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Conclusion(c)
Did you discuss any potential negative societal impacts of your work?
[No] We donot identify any potential negative societal impacts of minimizing number of trainableparameters in models.(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See appendix4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]