Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Driving
Anonymous Author(s)
AffiliationAddress email
Abstract
Aiming towards a holistic understanding of multiple downstream tasks simultaneously, there is a need for extracting features with better transferability.
Thoughmany latest self-supervised pre-training methods have achieved impressive performance on various vision tasks under the prevailing pretrain-finetune paradigm,their generalization capacity to multi-task learning scenarios is yet to be explored.
In this paper, we extensively investigate the transfer performance of various typesof self-supervised methods, e.g., MoCo and SimCLR, on three downstream tasks,including semantic segmentation, drivable area segmentation, and traffic objectdetection, on the large-scale driving dataset BDD100K. We surprisingly find thattheir performances are sub-optimal or even lag far behind the single-task baseline,which may be due to the distinctions of training objectives and architectural de- sign lied in the pretrain-finetune paradigm.
To overcome this dilemma as well as avoid redesigning the resource-intensive pre-training stage, we propose a simple yet effective pretrain-adapt-finetune paradigm for general multi-task training, where the off-the-shelf pretrained models can be effectively adapted withoutincreasing the training overhead.
During the adapt stage, we utilize learnablemulti-scale adapters to dynamically adjust the pretrained model weights supervised by multi-task objectives while leaving the pretrained knowledge untouched.
Furthermore, we regard the vision-language pre-training model CLIP as a strongcomplement to the pretrain-adapt-finetune paradigm and propose a novel adapternamed LV-Adapter, which incorporates language priors in the multi-task model viatask-specific prompting and alignment between visual and textual features.
Ourexperiments demonstrate that the adapt stage significantly improves the overallperformance of those off-the-shelf pretrained models and the contextual featuresgenerated by LV-Adapter are of general benefits for downstream tasks.
Introduction
Multi-task learning is a long-studied problem in computer vision and has become an emergingparadigm in autonomous driving [18, 45, 17].
An autonomous vehicle may need to concurrentlyperform a wide range of perception tasks, e.g., locating the pedestrians and cars, deciding roadaffordability for driving, as well as detecting the lanes to precipitate a driving action.
The jointlearning of multiple tasks can not only reduce the training and inference time, but also act as aregularizer that enforces the learning of generalizable representations [6, 18].
Apart from efficiencyand simplicity, an all-in-one architecture is argued to have the potential to enhance the robustness ofthe driving system by implicitly learning the synergy between heterogeneous tasks [18, 40, 35].
In the pursuit of the merits of multi-task learning, it is crucial to obtain universal features with greattransferability.
Recently, many concurrent self-supervised pre-training methods have shown great
[Figure]
potential when transferred to different types of vision tasks [42, 44, 39] under the pretrain-finetuneparadigm.
Despite the impressive performance, their transferability to multi-task learning scenarios isyet to be explored.
We argue that the heterogeneity of tasks will introduce a multitude of challenges totrain a unified model and it is often not the case that multi-task learning is of universal benefit.
Firstly,the widely adopted pretrain-finetune paradigm may lead to degraded performance in multi-tasklearning due to the misalignment of objectives between pre-training and fine-tuning [11, 15] sincemost supervised and self-supervised methods are highly specialized for a specific type of objectiveor task [3, 4, 42, 44].
Secondly, the performance of multi-task learning relies on many non-trivialfactors including model architectures, data augmentations, hyperparameters, convergence properties,etc [32, 34].
The specialized techniques catered for a specific type of architecture or task are largelynot applicable to a universal architecture since they are prone to fail during generalization.
Moreover,due to the labor-intensive process of data annotation, it is hard to collect complete annotations ofdifferent granularities for all tasks, which further complicates the situation.
In this paper, we focus on heterogeneous multi-task learning on partially labeled data (HMPL)
underthe realistic scenario of autonomous driving.
We first delve into the representation learning stage of
HMPL to reveal the performance degradation of different pre-training methods.
We thoroughly examine a wide range of pre-training methods including supervised pre-training (pretrained on ImageNet[31] ), classification-oriented methods (e.g., SimCLR [3], MoCo [14]), detection-oriented methods (e.g., DetCo [42]), segmentation-oriented methods (e.g., DenseCL [39]), and vision-languagepre-training methods (e.g., CLIP [28]) on three fine-grained tasks on the large-scale driving dataset
BDD100K [47], i.e., object detection, semantic segmentation, and drivable area segmentation.
Surprisingly, the performance of these methods varies greatly under the same training protocol, especiallyon the dense prediction task.
For example, MoCov2 [4] only gets 12.5 mIoU in semantic segmentationand segmentation-oriented models like DenseCL [39] and DetCo [42] perform unsatisfactorily ondense prediction tasks, which suggests that the misalignment lied in the pretrain-finetune paradigmcan lead to prominent performance degradation and there exists much room for improvement.
Given that ‘universal’ pre-training is still unsolved and redesigning the resource-intensive pre-trainingscheme comes with great computation overhead, we aim to develop a general approach that canfully harness the knowledge from the off-the-shelf pretrained models and make them amenableto multi-task scenarios via efficient adaptation.
We draw inspiration from the recent progress ofprompt-based learning in NLP [23, 50, 28, 10], where the language model pretrained on massiveamounts of raw text can be adapted to new scenarios with high efficiency by introducing hand-craftedor learnable prompts.
Following this philosophy, we propose a simple yet effective pretrain-adaptfinetune paradigm for multi-task transfer learning as a substitute for the dominating pretrain-finetuneparadigm in computer vision.
Concretely, during the adapt stage, learnable multi-scale adapterswith a small amount of parameters are tuned with frozen random initialized task-specific heads todynamically adapt the knowledge from the pretrained models under the multi-task objectives.
Albeitsimple, we show that the adapt stage mitigates the gap between pre-training and fine-tuning andsignificantly improves the overall performance across different pretrained models while being veryeffective and introducing no extra training overhead.
Apart from the supervised and self-supervised pre-training, Contrastive Language-Image Pre-training(CLIP) [28] manages to learn high-quality visual representation from an enormous amount of noisyimage-text pairs.
CLIP achieves breakthrough performance in zero-shot image recognition, whichindicates that it might have great potential for improving the generality and robustness of multi-tasklearning.
To this end, we propose LV-Adapter to further excavate the linguistic knowledge from CLIPand enhance the visual representations in the multi-task scenarios.
To maximize the correspondencebetween the dense features and the class-specific concepts, we first learn task-specific prompts inan end-to-end manner.
Then we model the language-to-vision adaptation function based on thecross-attention mechanism [38] to incorporate language priors in visual features.
[Figure]
Frozen parameters
[Figure]
We experimentally verify that our proposed pretrain-adapt-finetune paradigm significantly improvesthe overall performance of different off-the-shelf pre-training methods.
For example, it obtainsan absolute improvement by over 40% in semantic segmentation on MoCo-v1/v2, and DenseCL[14, 4, 39], and its superior performance does not rely on extra training costs or sophisticatedarchitectural design.
Furthermore, we showcase that our proposed LV-Adapter can serve as aneffective complementary approach for multi-task learning.
Especially, we experiment on threesettings that correspond to different levels of annotation quantity and observe that LV-Adapter bringsconsistent performance gains on three heterogeneous tasks simultaneously.
We hope our work canshed light on the heterogeneous multi-task learning by showing that effective adaptation is of greatessence to mitigate the deficiency in the dominating pretrain-finetune paradigm.
Related Work
Multi-Task Learning Multi-task learning aims to improve efficiency and accuracy through sharedinformation among multiple tasks.
MultiNet [36] joints classification, detection and semanticsegmentation via a unified architecture.
Eigen et al.
[8] address depth prediction, surface normalestimation, and semantic labeling.
YOLOP [40] leverages CSPDarknet as the backbone and adoptsthree specific decoders to solve object detection, drivable area segmentation, and lane detectionrespectively.
Standley et al.
[32] and Fifty et al.
[9] propose to identify proper task groupings formulti-task training instead of naively training all tasks together.
In this paper, we focus on developinggeneral and effective approaches for multi-task transfer learning in autonomous driving scenarios.
We provide more discussions with several related works in the Appendix.
Pre-training Methods A dominating paradigm in computer vision is to pretrain on a large scaleof data, e.g., ImageNet [31], then finetune on target tasks with usually less training data.
Recently,researchers are interested in learning visual representations without human supervision.
Classificationoriented methods typically rely on contrastive learning and online clustering, e.g., SimCLR [3], MoCo[14, 4], and BYOL [13].
Detection-oriented methods like DetCo [42] are specially designed forobject detection by conducting contrastive learning between the global image and local patches.
Segmentation-oriented methods like PixPro [44] further work on pixel-level correspondence tobenefit dense prediction downstream tasks.
We show that these methods are sub-optimal for multitask learning, and we focus on improving the performance of these off-the-shelf methods instead ofredesigning the computation-intensive pre-training stage.
Prompting Learning Natural language prompting freezes large-scale pretrained language modelsand reformulates text input with example or learnable prompts to bridge the gap between pre-trainingand model tuning efficiently.
GPT-3 [1] first introduces in-context learning, prepending hard textprompts for every task before the input.
Furthermore, recent methods inject learnable soft prompts[20, 25, 48] into the embedding space of the model and achieve impressive performance.
In the fieldof computer vision, CLIP [28] trains on millions of vision-language pairs and demonstrates effectivezero-shot classification.
CoOp [50], CPT [46], and CLIP-Adapter [10] further tunes fixed CLIP withsoft prompts by few-shot supervisions.
[Table]
Empirical Study
[Table] 3.1 Heterogeneous Multi-Task Setup
[Table]
We concentrate on three challenging heterogeneous tasks in autonomous driving, i.e., traffic objectdetection, semantic segmentation, and drivable area segmentation, on large-scale driving dataset
BDD100K [47].
We showcase the input and output of our model in Figure 1.
We select the state-ofthe-art detector and segmentation decoder head to build up our model.
To be specific, we follow thehard-parameter sharing scheme [6] where each task shares the same backbone and has its task-specifichead.
We outline our model architecture in Figure 2 and elaborate each component in the following.
Backbone & Neck.
We adopt the classic ResNet [16] as backbone and FPN [22] as neck to generatemulti-scale features.
The FPN neck [22] constructs pyramidal features, namely, { P 2 , P 3 , P 4 , P 5 } ,which have strides of { 4 , 8 , 16 , 32 } pixels w.r.t the input image and fixed 256 channels.
Segmentation Head.
We implement the decoder for segmentation based on MaskFormer [5].
MaskFormer formulates the segmentation task as a mask classification problem and its head iscomposed of a fully convolutional pixel decoder and a transformer decoder module.
The pixeldecoder takes { P 2 , P 3 , P 4 , P 5 } as input and gradually upsamples the features to produce highresolution per-pixel embeddings ε pixel .
The transformer decoder module utilizes a fixed set of queriesto attend to image features and produces mask embeddings ε mask .
Then ε pixel and ε mask is multipliedtogether to generate prediction masks.
We refer the reader to [5] for more details.
Detection Head.
We implement our detector based on Sparse R-CNN [33].
The detection pipelineconstructs a fixed set of learnable proposal boxes (e.g., 300) which serve as region proposals and arefed into a series of heads for prediction.
The dynamic instance interactive head dynamically generatesthe weights of a convolution filter that attends to the bins in the region proposals and predicts thelocation and category of object in a cascade manner.
We introduce three settings, i.e., the Disjoint-normal, Disjoint-balance, and Full-setting, corresponding to different levels of annotation quantity in our experiments.
For clarity, we defer the detaileddescriptions of three settings to Section 5.1.
For consistency, we conduct evaluations on the samevalidation set of BDD100K for different settings above.
We mainly work under the Disjoint-normalsetting for comparisons between the performance of different pre-training methods as well as ablationstudies.
And we further verify the efficacy of our proposed methods on the Disjoint-balance and the
Full-setting in Section 5.2.
3.2 Multi-Task Learning Baselines
Self-training Based on the available annotations for each task, we trained three single-task teachermodels on the labeled data and use them to generate pseudo labels on unlabeled data following[43, 11].
For single-task teacher models, we adopt the same architecture and training schedules of
Sparse R-CNN [33] and MaskFormer [5].
For object detection, we use a score threshold of 0.5 toselect pseudo box labels.
For segmentation, we use the score threshold of 0.3 to select segmentationmasks.
Pixels with lower scores are assigned with the ignore label.
Thereafter, every image hasannotations for all tasks and we train a multi-task student model with a weighted sum of all objectivesfor each task:
[Equation]
where L det , L sem , L driv are losses for object detection, semantic segmentation, and drivable areasegmentation, respectively.
1 We include more details in Section 5.1.
[Table]
Vision-language
CLIP [28] 54.
Zeroing loss Some works [18, 41] simply zero losses for a particular task if the input image does nothave the corresponding annotation.
Batch-level Round-Robin
In this scheme [47, 21], each batch consists of samples that have the sametype of annotations for a task and different tasks are repeated in a fixed order during training.
Task Scheduling This is a stochastic schedule that the task for each SGD update step is sampledfrom a distribution [21].
We mainly consider a uniform task sampling schedule that sampled from auniform distribution, and a weighted task sampling schedule where the sampling weight for a task isproportional to the number of labeled images and training epochs for this task.
Performance Comparison As can be seen in Table 1 and Table 3, these methods surpass single-taskteacher model by a clear margin, which indicates that the model benefits from multi-task learning.
Among these baselines, self-training achieves satisfactory performance while being conceptuallysimple, thus we adopt it for the rest experiments in this paper.
3.3 Pretrain-finetune for Multi-Task Learning
In this section, we adopt the identical training protocol and architecture to extensively investigatethe performances of different types of pre-training methods, including task-oriented methods andvision-language pre-training, when transferred to the multi-task learning scenario.
We revisit thetechnical details of these different schemes in the Appendix.
We report the performance of theaforementioned methods in Table 2, and come to the following observations:
• Only SimCLR achieves decent performance on all three tasks and many methods encounter substantial degradation on pixel-level segmentation tasks.
For semantic segmentation, MoCo- v1/v2 and DenseCL achieve the worst mIoU (degrade by 66%-83% v.s. ImageNet-pretrained).For drivable area segmentation, MoCo-v1/v2, DenseCL, and CLIP have the worst performance(degrade by nearly 12% v.s. ImageNet-pretrained).• The pre-training paradigm does not seem to have an explicit correlation with the downstreamperformance, and even the task-oriented design in pre-training does not guarantee the transferperformance on the corresponding task type. For example, the segmentation-oriented DenseCLachieves sub-optimal performance on two segmentation tasks.
Image
Encoder
Text
Encoder v !!
v "!
… v #! v !" v "" … v #" v !$ v "$ … v #$ personcar roadbuilding direct alternative
T
Head #
Head # Head #
Teacher # Teacher #
Teacher #
GT labels
Pseudo labels
Transformer
Decoder key & value query …
P s e ud o l a b e li n g
LV-Adapter … … …
Effective Adaptation for Multi-Task Learning
[Figure]
4.1 Pretrain, Adapt, then Finetune
[Figure]
We attribute the degraded transferring performance of the state-of-the-art pre-training methods totwo critical factors, i.e., optimization gap and architectural gap.
Firstly, the high specialization ofthe pre-training methods and the heterogeneity of downstream tasks result in the optimization gapin the classic pretrain-finetune paradigm.
To be specific, the objective for pre-training is usually atype of contrastive loss (e.g., InfoNCE [37]) while the fine-tuning stage is supervised by a weightedsum of several task-specific losses.
Secondly, we notice that many models in Table 2 are pretrainedwith a convolutional head, e.g., Fast R-CNN detector [12] or convolution-based projection head.
Incontrast, we adopt the transformer-based MaskFormer head for segmentation tasks.
The distinctionsbetween convolution and transformer lie in the inductive bias (local v.s. global) and the internalrepresentation structure [29] and we conjecture this plays a non-negligible role in transferring.
Wefurther experimentally verify this point in Appendix.
The limited transfer performance presented in Table 2 indicates that the ‘universal’ pre-trainingremains unsolved.
Instead of redesigning another pre-training scheme that brings vast computationoverhead, we aim to efficiently reuse the intact knowledge of these off-the-shelf pre-trained modelswith minimal modifications to the architecture.
We draw inspiration from the recent progress ofprompt-based learning, where prompt tuning emerges as a new alternative to fine-tuning.
Forexample, P-Tuning v2 [24] matches the fine-tuning performance by only tuning learnable promptswhile freezing the large-scale language model.
Following this philosophy, we propose a simpleyet effective pretrain-adapt-finetune paradigm to mitigate the gap between the pre-training andfine-tuning stage.
During the adaptation stage, we are given the pretrained model weights (e.g.,
ResNet-50) inherited from the pre-training stage.
We aim to transform the model weights via asmall amount of learnable parameters to adapt the knowledge of the pretrained weights towardsmulti-task scenarios.
To this end, we freeze the parameters of the random initialized task-specificheads and the backbone, and tune the parameters of Feature Pyramid Network (FPN) supervisedby the multi-task loss function in Equation 1.
During fine-tuning , all parameters are activated andupdated via gradient descent.
We compare different schemes for multi-task learning in Figure 2.
This adaptation stage characterizes several critical design choices: (1) This stage bridges the gapbetween pre-training and fine-tuning by including the pretrained weights and multi-task objectivessimultaneously, and the frozen backbone prevents the pretrained weights from being spoiled beforethe fine-tuning stage.
Compared to the single-layer prompt/adapter in P-Tuning [19] and CLIPAdapter [10], the FPN acts as a multi-scale adapter that is endowed with greater capacity and providessemantically stronger features for fine-grained downstream tasks.
(3) Taking FPN as the adaptermeans that the model architecture for the pretrain-finetune and pretrain-adapt-finetune paradigmis identical and effective adaptation can be achieved without any modification to the architecture.
We have experimented with more sophisticated adapters (e.g., scale-aware adapters, lightweighttransformers) and observed marginally better results.
Hence, we opt for the original architecture above for simplicity.
We experimentally verify that our pretrain-adapt-finetune paradigm significantlyimproves the performance and stability of different kinds of pre-training methods (c.f. Section 5.2)without increasing the total training costs (c.f. Section 5.3).
4.2 Language-to-Vision Adapter
From the perspective of transfer learning, CLIP can serve as a complementary scheme to enhancethe pretrain-adapt-finetune paradigm, since it can comprehend the concepts in natural language.
Basically, the CLIP model excels in aligning the visual and language embeddings and some works[49, 30] claim that the textual features generated by CLIP have meaningful correspondence to thesemantic regions in an image.
We note that the result of CLIP in Table 2 only reuses the weightsof its image encoder and discards the text encoder.
Therefore, we take a step further to explicitlyexploit the knowledge in the full CLIP model.
We pursue to underpin the compatibility between thesemantic concepts of each task and the image features and generate semantically stronger contexts fordownstream tasks.
The resulting model, named LV-Adapter, is outlined in Figure 3, and we elaborateon each ingredient in the followings.
The CLIP model adopts ResNet [16] and BERT [7] as the encoder for image and text, respectively.
Formally, we denote the last output feature maps from ResNet stages as { x i } 5 i =2 .
Attentionalpooling and L 2 normalization are applied to x 5 ∈ R H 5 W 5 × C 5 to produce the global image featureˆ I e ∈ R 1 × C 5 for zero-shot inference, which can be formulated as follows:
[Equation]
where GAP( · ) denotes global average pooling, MHSA( · ) denotes multi-head self-attention [38], and⊕ denotes concatenation operation.
To extract textual features of each class, class-specific promptsare constructed via a generator function and are fed into the text encoder.
We denote the normalizedoutput features for N classes as ˆ T e ∈ R N × C , which can be obtained as follow:
[Equation]
where Gen( · ) is the generator function of class-specific prompts, TE is the text encoder, and { n i } Ni =1is
the class name embeddings.
Though ˆ T e and ˆ I e are well aligned in the image-text contrastive pretraining, they do not preserve any spatial details and are not readily applicable for downstream visiontasks.
In contrast, the multi-level features output by FPN are semantically rich in the spatial dimensionbut are not directly aligned in the pre-training stage.
Therefore, we pursue to learn task-specificprompts and propose a Language-to-Vision adapter to enhance the pixel-class correspondence.
Learning Task-Specific Prompts.
The original generator function of CLIP outputs hand-craftedprompts using the predefined template, e.g., “ a photo of a [CLS] .”.
However, the performance ishighly sensitive to the form of the template [28, 50], which may be sub-optimal when transferring toheterogeneous downstream tasks.
Hence, we follow CoOp [50] to use learnable textual contexts inprompting for each task.
In spite of the inconsistent output formats of the three tasks (4-D box v.s.dense output), they both need to determine the category of boxes or pixels, and we incorporate theclass names of each task in prompting.
The task-specific prompting can be formulated as follows:
[Equation]
where n i refers to the embeddings of the class name belonging to a specific task, and v t is thetask-specific learnable contexts and the superscript indicates the task type.
Enhancing Pixel-Class Correspondence.
To align the textural features and the dense FPN features,we propose a Language-to-Vision Adapter to incorporate the language priors into the visual features.
Formally, we denote the last feature map of P 5 as z 5 ∈ R H 5 W 5 × C , and we aim to learn an adapterfunction A L → V to generate language-aware contexts for downstream tasks.
We utilize the crossattention mechanism in Transformer decoder [38] for Language-to-Vision adaptation:
[Equation]
where the q, k, v stands for query, key, and value.
A single linear fully connected layer is used toadjust the channel number of ˆ T e and we omit it in Equation 5 for simplicity.
We denote the output of
Equation 5 as ˜ z 5 ∈ R H 5 W 5 × C .
After adaptation, we simply substitute z 5 with ˜ z
5 to inject linguisticknowledge into the high-level features of FPN and leave the task-specific head design unchanged. We
[Table]
follow the same paradigm described in Section 4.1 when incorporating LV-Adapter: only the adaptermodule (i.e., LV-Adapter and FPN) is tuned during the adaptation stage.
Moreover, we freeze theweights of the CLIP text encoder throughout the paper to preserve the language priors.
We verify theefficacy of our LV-Adapter and disentangle the effect of different modules ablation in Section 5.3.
Experiments
5.1 Experiment Settings
In BDD100K, 70k training images are labeled for both object detection and drivable area segmentation,and only 7k training images are labeled for semantic segmentation.
These two sets are not disjoint andabout 3k images have complete annotations for all three tasks.
As is described in [11], data annotationis one of the biggest challenges of training a multi-task model.
In real-world scenarios, it is unrealisticto obtain complete types of annotations for all input images, especially in the traffic scene wherefine-grained annotations are required.
In this paper, we are interested in the performance of multi-tasklearning with different quantities of annotations.
Thus, we mainly consider three scenarios thatcorrespond to three different levels of annotation scarcity, namely, Disjoint-normal, Disjoint-balance,and Full setting.
We mainly work under the Disjoint-normal setting where the annotations of eachtask are disjoint, and the number of labeled images for each task is in decreasing order as follows:drivable area segmentation (20k), object detection (10k), semantic segmentation (7k).
We provide thedetails of these three settings in the Appendix.
Implementation Details.
The training epoch is fixed as 36, syncBN [27] is on, the learning rateat the fine-tuning stage is 2 .
5 × 10 − 5 , and weight decay is 1 × 10 − 4 .
The image scale is 1280 ×(720, 600).
No other data augmentation is used.
We adopt the AdamW [26] optimizer.
The warmupiteration is 1000 and the warmup factor is 0.01.
For LV-Adapter, the learnable prompts are prependedto the class, and the length of prompts is 16.
During the adaptation stage, the learning rate is set to2 .
5 × 10 − 4 .
The layer of transformer decoder in Equation 5 is 3.
All experiments are conducted onservers with 8 Nvidia V100 GPU (32GB) cards and Intel Xeon Platinum 8168 CPU (2.70GHz).
5.2 Main Results
We present the results of our proposed paradigm in Table 2.
As can be seen, with the conventionalpretrain-finetune paradigm, the performances of the state-of-the-art self-supervised methods are unstable, especially in semantic segmentation and drivable area segmentation.
For example, MoCo-v2only achieves 10.3 mIoU on semantic segmentation.
In contrast, under the pretrain-adapt-finetuneparadigm , MoCo-v2 improves by a large margin in semantic segmentation ( + 50.9 in mIoU).
Notably, MoCo-v1/v2 and DenseCL obtain an absolute improvement by over 40% mIoU in semantic segmentation.
The results demonstrate that the pretrain-adapt-finetune paradigm significantly improves the
[Table] [Table]
overall performance across different types of pretrained models.
Moreover, we further showcase theefficiency of this paradigm in Section 5.3.
We compare the results of our LV-Adapter to single-taskbaselines and the self-training baseline using the ImageNet pretrained model under three settings in Table 3.
It demonstrates that LV-Adapter performs better on all metrics.
Therefore, we conclude thatthe language-guided pretrain-adapt-finetune paradigm can effectively reduce the gap between thepre-training and fine-tuning stage by incorporating the linguistic knowledge to visual features.
5.3 Ablation Study
Module design.
We compare the performance of different module designs in Table 4 using thepre-trained weights from CLIP.
Row #1 refers to the performance of the pretrain-finetune paradigmusing the pretrained weights of the CLIP image encoder.
Row #2-#5 refer to the performance of thepretrain-adapt-finetune paradigm and row #3-#5 reuse the pretrained weights of CLIP text encoder.
The Prompt column refers to naive prompting without alignment between visual and textual features[30]: we compute class-specific textual features by prompting the CLIP text encoder (Equation 4)and compute the class activation maps by computing dot product between the textual features and z 5after normalization, and the output maps are concatenated to z 5 and a 1 × 1 convolution layer is usedto reduce the number of channel.
The V2L column refers to the post-model module in [30], wherethe image context is used to prompt the language model and language-compatible class activationmaps are concatenated back to z 5 .
In comparison, our Language-to-Vision Adapter (L2V column)directly incorporates linguistic knowledge into visual features and does not rely on dense activationmaps and is thus more lightweight.
As can be seen, when retrieving knowledge from pretrained textencoder via continuous prompt, the model improves in semantic segmentation by 0.3 mIoU (#2 vs#3).
When we adopt the Vision-to-Language Adapter to update the text features, the performancedegrades slightly in semantic segmentation (#3 vs #4).
In contrast, our proposed LV-Adapter canbetter extract prior knowledge from CLIP and achieve the best results ( + 0.4 and + 0.2 in mIoU insemantic segmentation and drivable area segmentation respectively).
Number of epochs of the adapt and finetune stage.
We compare different configurations of adaptand finetune epochs in Table 5.
It shows that the performance improves when the adapt stage isincluded. When adapting with fewer epochs, the performance improves gradually and the bestconfiguration is 1 epoch for adapt and 35 epochs for finetune .
Since we freeze the backbone andrandomly initialized heads during the adapt stage, the number of trainable parameters is moderate. Hence these parameters can be adapted within a few iterations in the adapt stage.
Conclusion
In this paper, we present the first effort on revealing the degradation of state-of-the-art self-supervisedmodels under the multi-task learning scenario in autonomous driving.
To reduce the gap between thepre-training and fine-tuning stage, we propose a simple yet highly efficient pretrain-adapt-finetuneparadigm, which boosts the performances of different self-supervised pretrained models by a largemargin without increasing the overall training overhead.
We further utilize the vision-languagepre-training model CLIP as a complement to our proposed paradigm and propose LV-Adapter, whichincorporates linguistic knowledge into visual features via learning task-specific prompts in a novelfashion.
Extensive experiments showcase that (1) the adapt stage is the key to mitigate the deficiencyof the dominating pretrain-finetune paradigm in multi-task learning, and (2) the language priorsencoded by CLIP are of general benefits for multiple downstream tasks.