M 3 ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design
Abstract
Multi-task learning (MTL) encapsulates multiple learned tasks in a single modeland often lets those tasks learn better jointly.
Multi-tasking models have becomesuccessful and often essential for many sophisticated systems such as autonomousdriving and indoor robots.
However, when deploying MTL onto those real-worldsystems that are often resource-constrained or latency-sensitive, two prominentchallenges arise: (i) during training , simultaneously optimizing all tasks is oftendifficult due to gradient conflicts across tasks, and the challenge is amplified whena growing number of tasks have to be squeezed into one compact model; (ii) atinference , current MTL regimes have to activate nearly the entire model even tojust execute a single task.
Yet most real systems demand only one or two tasks ateach moment, while flexibly switching between tasks per need: therefore such “alltasks activated” inference is also highly inefficient and non-scalable in practice.
In this paper, we present a model-accelerator co-design framework to enable efficient on-device MTL, that tackles both training and inference bottlenecks.
Ourframework, dubbed M 3 ViT, customizes mixture-of-experts (MoE) layers into avision transformer (ViT) backbone for MTL, and sparsely activates task-specificexperts during training, which effectively disentangles the parameter spaces toavoid different tasks’ training conflicts.
Then at inference with any task of interest,the same design allows for activating only the task-corresponding sparse “expert”pathway, instead of the full model.
Our new model design is further enhanced byhardware-level innovations, in particular, a novel computation reordering schemetailored for memory-constrained MTL that achieves zero-overhead switching between tasks and can scale to any number of experts.
Extensive experiments on
PASCAL-Context [1] and NYUD-v2 [2] datasets at both software and hardwarelevels are conducted to demonstrate the effectiveness of the proposed design.
Whenexecuting the practical scenario of single-task inference,
M
3 ViT achieves higheraccuracies than encoder-focused MTL methods, while significantly reducing 88% inference FLOPs.
When implemented on a hardware platform of one Xilinx
ZCU104 FPGA, our co-design framework reduces the memory requirement by2.40 × , while achieving energy efficiency (as the product of latency and power) upto 9.23 × higher than a comparable FPGA baseline.
Code will be released.
Introduction
Vision Transformers (ViTs) [3, 4, 5, 6], as the latest performant deep models, have achieved impressiveperformance on various computer vision tasks [7, 8, 9].
These models are specially trained or testedfor only one or a few tasks; however, many real-world applications require one compact system that can handle many different tasks efficiently, and often need to swiftly switch between tasks perdemand.
For example, an autonomous driving system [10] needs to perform and switch between manytasks such as drivable area estimation, lane detection, pedestrian detection, and scene classification:apparently both single task inference and cross-task switching need to happen at ultra-low latency.
As another example, smart-home indoor robots [11] are expected to address semantic segmentation,navigation, tracking or other tasks in varying contexts, with very limited on-board resources.
Multitask learning (MTL) [12, 13, 14] solves multiple tasks simultaneously within a single model and learnsimproved feature representations [15] shared by related tasks [16, 17].
Therefore, accomplishingrealistic efficient MTL is becoming a key knob for building real-time sophisticated AI systems.
Despite the promise, challenges persist to build an efficient MTL model suitable for real worldapplications: ➊ during training , prior works [18, 19, 20] indicate the competition of different tasksin training may degrade MTL, since the same weights might receive and be confused by conflictingupdate directions.
Specifically, [19] reveals that negative cosine similarities between different tasks’gradients are detrimental.
[21, 22] confirm that conflicting gradients not only slow down convergencebut also bias the learned representations against some tasks.
That is only getting worse on compactmodels owing to their limited modeling capacity.
To tackle the cross-task conflicts, solutions havebeen proposed by varying learning rate speeds of different tasks [20], using “cross-stitch” sharing [23],or re-balancing task gradients [19, 24, 20, 25].
However, they either require task-specific design orsignificantly increase the model complexity which contradicts our efficiency goal.
➋ at inference ,existing MTL regimes typically activate the entire backbone model unconditionally.
However, manyreal systems only need to call upon one or a few tasks at each moment , hence the “all activated”inference is heavily inefficient and non-scalable.
For example, current regimes [14, 23, 26, 27] haveto activate the whole gigantic ResNet [28] encoder even just to execute a single monocular depthestimation task or so.
If the number of tasks scale up [29] and the backbone keeps growing bigger,the “per task” inference efficiency of the resultant MTL model could become catastrophically poor.
To tackle these bottlenecks, we propose a model-accelerator co-design framework that enablesefficient on-device MTL.
Specifically, in the software level, we propose to adapt mixture of experts(MoE) layers [30, 31] into the MTL backbone, as MoE can adaptively divide-and-conquer the entiremodel capacity into smaller sub-models [30, 32].
Here, we replace the dense feed-forward-networkin the ViT with sparsely activated MoE experts (MLPs).
A task-dependent gating network will betrained to select the subset of experts for each input token, conditioning on tasks.
During training,this task-dependent routing principle effectively disentangles the parameter spaces, balancing featurereuse and automatically avoiding different tasks’ training conflicts.
Meanwhile, at the inferencestage with any task of interest, this design naturally allows for sparse activation of only the expertscorresponding to the task instead of the full model, thus achieving highly sparse and efficient inferencefor the specific task.
In the hardware level, we propose a novel computation reordering mechanismtailored for memory-constrained MTL and MoE, which allows scaling up to any number of expertsand also achieves zero-overhead switching between tasks.
Specifically, based on ViT, we push tokensto per-expert queues to enable expert-by-expert computation rather than token-by-token.
We thenimplement a double-buffered computation strategy that hides the memory access latency required toload each experts’ weights from off-chip memory, regardless of task-specific expert selection.
Thisdesign naturally incurs no overhead for switching between frames or tasks in FPGA.
To validate the effectiveness, we evaluate our performance gain using the ViT-small backbone onthe NYUD-v2 and PASCAL-Context datasets.
On the NYUD-v2 dataset with two tasks, our modelachieves comparable results with encoder-focused MTL methods while reducing 71% FLOPs forsingle-task execution.
When we evaluate on the PASCAL-Context dataset with more tasks, our modelachieves even better performance (2.71 vs. 0.60) and reduces 88% inference FLOPs.
We found the
MTL performance gain brought by MoE layers consistently increases as the task count grows.
Whenimplemented on a hardware platform of one Xilinx ZCU104 FPGA, our co-design framework reducesthe memory requirement by 2.40 × while achieving energy efficiency (as the product of latency andpower) up to 9.23 × higher than comparable FPGA baselines and up to 10.79 × higher than the GPUimplementation.
Our contributions are outlined below:
• We target the problem of efficient MTL, and adopt the more realistic inference setting
(activating one task a time, while switching between tasks).
We introduce MoE as theunified tool to attain two goals: both resolving cross-task training conflicts ( better MTLperformance ), and sparsely activating paths for single-task inference ( better efficiency ).Specifically
for MTL, the MoE layer is accompanied with a task-dependent gating networkto make expert selections conditioning on the current task.
• We implement the proposed MTL MoE ViT framework on a hardware platform of one XilinxZCU104 FPGA, which enables us to exploit a memory-efficient computation reorderingscheme that consolidates per-expert Multiply-and-ACcumulate (MAC) operations such thatonly one expert
’s weights are needed on-chip at a time.
Our design is scalable to any numberof experts while requiring no frame-switching or task-switching overhead.•
We conduct extensive experiments to justify its inference effectiveness in both accuracy andon-edge efficiency metrics.
Our framework, dubbed M 3 ViT, achieves higher accuracies thanencoder-focused MTL methods, while significantly reducing 88% inference FLOPs; onhardware, it reduces the memory requirement by 2.40 × and costs up to 9.23 × and 10.79 ×less energy compared to the FPGA and GPU baselines, respectively.
Related Works
Multi-task Learning
The generic multi-task learning problem has been studied for a long history.
Some non-deep learning-based methods propose to use distance metric [33, 34, 35], probabilisticprior [36, 37, 38, 39, 40] to model the common information among tasks.
With the emergence of thedeep learning technique, MTL is performed to learn shared representation among tasks.
The emergingof ViT further makes it possible to extent the task range from only vision tasks to other modalitiestasks (e.g., text, audio) [41, 42, 43, 44, 45].
Current MTL models can be roughly categorized intotwo types based on where the task interactions take place in the network.
The encoder-focusedarchitectures [23, 46, 26, 27] only share information in the encoder, before decoding each task withan independent task-specific head.
Cross-stitch networks [23] introduce linear combination in eachlayer.
NDDR-CNN [26] improves it by dimensional reduction.
MTAN [27] leverages an attentionmechanism to learn between tasks.
The second type, decoder-focused models [47, 48, 49, 50], makeinitial task predictions in decoder and then leverage features from these initial predictions to furtherimprove output.
Although they report higher performance, their models consume a large numberof FLOPs [14].
This makes it difficult to deploy them onto those real-world systems that are oftenresource-constrained or latency-sensitive.
And they need to execute all the tasks for initial prediction,which is heavily inefficient in the common scenario when only one or few tasks are needed.
Hence, wefocus on encoder-focused architecture in this work.
Moreover, some other methods [25, 20, 51, 27]are proposed to handle the multi-task training conflicts problem.
Mixture of Experts (MoE) MoE contains a series of sub-models (i.e., experts) and performsconditional computation in an input-dependent fashion [52, 53, 54, 55, 56], based on learned ordeterministic routing policies [57, 56].
The traditional dense MoEs suffer from intensive computational costs since they select all experts [58].
Recent studies [30, 59, 60] in natural languageprocessing (NLP) propose sparse MoE that sparsely activates a few experts during both trainingand inference, thus substantially reducing the cost and allowing gigantic language models evenwith trillions of parameters [60].
Unfortunately, such a sparse-gated manner still has limitations ofunstable training and imbalanced selections among experts.
Various solutions are invented from regularization [61, 59, 60] and optimization [62, 63] perspectives.
Moreover, MoE has drawn increasingpopularity in computer vision [58, 64, 65, 66, 67, 68, 69], where it mainly focuses on considerablysmaller network backbones compared to the ones in NLP.
For instance, [66] and [67] formulate thechannel and kernel of convolutional layers as experts and establish the MoE framework.
Severalpioneer investigations also explore MoE for multi-task learning, which are related to this work.
Particularly, [17, 70, 71] introduce task-specific gating networks to choose different parts of modelsfor processing information from each task.
They present certain possibilities of using MoE to solve
MTL problems in some cases like classification for medical signals [70], digital number images(MNIST)
[71], and recommendation systems [17].
We make a further attempt to adapt MoE into acompact model for dense prediction multi-task learning, along with software-hardware co-design.
Hardware FPGA acceleration of Transformer-based models has attracted increasing attention.
Works such as VAQF [72], SPViT [73], FTRANS [74], Qi et al.
[75], and Peng et al.
[76] notethat Transformer models are computation- and memory-intensive and are too large to fit on the
FPGA.
Therefore, various model compression methods have been proposed, such as activationquantization, token pruning, block-circulant matrices (BCM) for weights, block-balanced weightpruning, and column-balanced block weight pruning.
Such compression methods are lossy andrequire compression-aware training to regain accuracy.
To the best of our knowledge, there is noexisting FPGA accelerator for MoE in a Transformer-based model.
The MoE mechanism exposesgreat challenges to FPGA since it requires swift expert switching between tokens and frames, which may introduce significant overhead of memory and parameter loading.
In this work, however, we
[Figure]
propose a novel expert-by-expert computation-reordering approach that can reduce the overhead to
[Figure]
negligible despite the number of experts, and does not require model compression or re-training.
[Figure]
Method
[Figure]
Overview We first describe the standard Vision Transformer and MoEs, and then show the proposed
MoE ViT design for MTL.
To enable dynamically adapting between different tasks with minimumoverhead on FPGA, we detail the hardware implementation.
Figure 1 shows the whole framework.
[Figure] [Figure]
3.1 Task-dependent MoE ViT Design
Vision Transformer
The representative Vision Transformer architecture [3] first splits the inputimage into non-overlapped patches and projects the patches to a higher hidden dimension using oneconvolutional layer.
The projected patches (a.k.a.
tokens) are then passed through several consecutivetransformer layers.
Each layer contains a self-attention module and a feed-forward-network (MLPs).
The self-attention is computed using the scaled-dot product:
[Equation]
where Q , K , V ∈ R N × C are the query, key and value matrices computed from input tokens; N and
C indicate the token number and the hidden dimension.
In our experiments, we adopt the DeiT [4] asthe backbone encoder, which is a data-efficient ViT variant that distills tokens to ensure the studentlearns from the teacher through attention.
Mixture of Experts Layer A Mixture of Experts (MoE) layer typically consists a group of Nexperts f 1 , f 2 , · · · , f N along with a router R (or gating network) to select the corresponding experts.
The experts network can be several layers of convolutional filters [77] in CNNs or multi-layerperceptrons [60, 78] in ViTs.
The router R plays a key role within our MoE ViT design as itdetermines task routings via only sparsely activating relevant experts.
We adopt a representativerouter called top- K gating [30] based on ViT. With input x , the resultant output of MoE layers can beformulated as the summation of the selected top K experts from N expert candidates using a router:
[Equation]
if v is in the top K elements
[Equation]
where G represents the learnable network within the router, for which we employ a single-layer MLPin practice.
The softmax( · ) together with TopK( · , K ) sets all elements of the vector to zero exceptthe elements with the largest K values.
In practice, we choose K = 4 out of N = 16 expert candidates.
Each expert is computed with W 2 σ gelu ( W 1 x ) , where σ gelu is the GELU activation [79].
W 1 and W 2are two learnable weight matrix.
Note that we scale down the expert size by four times compared tothat in standard ViT MLP layers to make the computation FLOPs equivalent.
We also employ theload and important balancing loss with weight of 0.01 following [30] to avoid always picking thesame experts while ignoring others.
This loss term is also employed for the two task-dependent MTL
MoE designs that we introduce next.
Multi-gate MoE ViT for MTL MoE brings training dynamics to balance between large capacityand efficiency, by selecting only a subset of experts using the router.
To adapt vanilla MoE into ourdense prediction MTL framework, we first propose to assign each dense prediction task a router R ito specify its own experts, denoted as multi-gate MTL MoE ViT:
[Equation]
5) where i denotes task index.
Expert candidates f k are shared across tasks.
The flow chart of themulti-gate variant is shown in Figure 2(a); task-dependent routers take as input the shared tokenembedding and do their expert selections.
Task-conditioned MoE ViT for MTL Conditional encoding has been widely applied to multimodal [80] and multi-task [81] models.
To achieve task-dependent routing with one gating network,we propose the task-conditioned MTL MoE ViT shown in Figure 2(b).
Specifically, suppose we haven tasks in training.
We manually define a n -dimensional one-hot task-conditioned vector.
The vectoris fed into a two-layer MLP to extract a 64-dimensional task embedding, which is then concatenatedwith token embeddings to form the task-dependent input for the router in the MoE layer:
[Equation]
7) where T indicates the two-layer MLPs to extract task-conditioned embeddings, e i ∈ { 0 , 1 } n , and 196 (cid:80) nj =1 e j = 1 .
We denote this conditional design as task-conditioned MTL MoE ViT, in whichbackbone model parameters do not proportionally increase if we include more tasks in training.
3.2 Circuit-level Implementation
We co-design the hardware to support
MTL MoE ViT.
We design a layer-wise implementation of
M 3 ViT on a Xilinx ZCU104 FPGA, a diagram of which is provided in the supplementary material.
The design computes layers sequentially but parallelizes computation steps within each layer.
Byproposing a novel computation reordering scheme, our hardware design features memory-efficientexpert computation that also achieves zero-overhead task switching and frame switching.
Challenges of Naive Method A straightforward (but naive) implementation would compute theoutput for each token in the order it appears in the input sequence: all tokens choose any K expertsout of the N candidates, so ostensibly the only way to avoid data loading overhead would be to keepweights of all N experts on-chip at all times.
However, this requires extreme on-chip memory usage,scaling with O ( N ) and typically exceeding FPGA on-chip memory capacity unless N is very small.
Challenges of Cache-based Method We can adopt a cache to store several experts on-chip at anygiven time.
However, this on-demand approach incurs long delays from off-chip DRAM accesseswhenever the cache needs to be repopulated with an expert’s weights.
Further, we experimentallyfound that all experts are likely to be activated at least once across all tokens, exhibiting a cacheunfriendly access pattern.
Therefore, although a cache-based design alleviates memory inefficiency,it incurs severe delays by frequently loading the weights of experts.
Proposed Solution: Memory-efficient Computation Reordering
The crux of the problem lies inthe unpredictability of the set of experts that will be needed by tokens at any given time.
We addressthis problem at its root by designing a novel computation reordering scheme that flips the computepattern on its head: rather than computing the MoE layer token-by-token, we instead compute itexpert-by-expert.
The overall flow chart of the reordering scheme is shown in Figure 3.
[Figure]
buffering [82]: one buffer is filled withan expert’s weights from off-chip memory accesses, while another already-loaded buffer is used tocompute another expert’s results for its entire token queue.
After both operations finish, the buffersare swapped, and the process repeats.
Scalability and Efficiency Our approach hides nearly all latency from off-chip memory accesses toload expert weights, and it uses O (1) on-chip memory with respect to K and N , making it scalableto any number of experts.
Additionally, our method’s efficiency does not rely on any specific usagepattern of experts for a given frame or a given task, so we naturally achieve zero-overhead switchesbetween frames and between tasks.
Task switches and frame switches in our hardware design do notchange our computation flow at all, and there is no specific step taken to execute the switch.
Experiments
4.1 Experiment Setup
To evaluate the propose method, we conduct experiments on two popular dense labeling MTLbenchmarks, i.e. NYUD-v2 [2] and PASCAL-Context [1].
Both datasets are described below.
Datasets The PASCAL-Context [1] contains a total of 10,103 images, for the five tasks of edgedetection (Edge), semantic segmentation (Seg.), human parts segmentation (H.Parts), surface normals(Norm.), and saliency detection (Sal.).
The NYUD-v2 dataset [2] is an indoor dataset which consistsof RGB-D images of 464 indoor scenes.
There are 795 images for training and 654 images for testing,both with annotation for semantic segmentation (Seg.) and monocular depth estimation (Depth).
Evaluation Metrics For software level evaluation, we adopt the standard evaluation metricsfollowing [14, 83, 49].
Particularly, we use mean intersection over union (mIoU) for semanticsegmentation, human parts segmentation, and saliency; mean error (mErr) for surface normalsestimation, root mean square error (rmse) for depth estimation; and optimal dataset F-measure (odsF)[84] for edge detection.
Following [14], we use ∆ m to evaluate a MTL model m as the average pertask drop with respect to the STL model b over all tasks:
∆ m = 1 T (cid:80) Ti ( − 1) l i ( M m,
i − M b,i ) /M
b,i ,where M m,i and M b,i are the metrics of task i for the model m and b respectively, and l i = 1 if alower value means better performance.
[Table]
68.8 −1.
M 2 ViT (+MoE)
MoE ViT-small 72.
71.7 +2.
M 3 ViT (+MoE+Codesign) MoE ViT-small 72.
71.7 +2.
M T L - B
M - V i T
M 3 - V i T
M T L - B
M - V i T
M 3 - V i T
To evaluate our model-accelerator design, we consider latency, energy usage (as the product of latencyand power), and on-chip memory usage for single-task inference using a batch size of 1.
Network Configuration and Implementation Details We evaluate our model based on severalversions of ViT backbone including ViT-tiny, ViT-small, and ViT-base.
Our FPGA designs target the
Xilinx ZCU104 FPGA at a 300 MHz clock frequency, consuming 10 W of power.
The GPU baselinesare measured on the NVIDIA Quadro RTX 8000.
The results reported below are based on ViT-small.
Please refer to the supplement for training setup, more details on network configuration, results on
ViT-tiny and ViT-base, and details of our target hardware platforms.
[Table]
[Table] [Table]
4.2 Comparison with State-of-the-art Dense Prediction MTL
As we target an efficient MTL system under the single-task inference setting
[Table]
we conduct experimentson encoder-focused architectures (more details in Section 2).
MTL-B [14] is a vanilla multi-tasklearning baseline model which is composed of a shared backbone in combination with task-specificheads.
Several state-of-the-art (SoTA) encoder-focused MTL models, including MTAN [27], CrossStitch [23] and NDDR-CNN [26], improve MTL-B by proposing feature sharing methods in theencoder.
Our methods are all conducted on vanilla MTL-B, namely, applying MTL on ViT (M-ViT),adding task-dependent MoE design (M 2 ViT), and adding hardware co-design on FPGA (M 3 ViT).
We also compare with previous works that handle the multi-task training conflicts problem, includinguncertainty weighting [25], GradNorm [20], DWA [51], and MGDA [27], and they are evaluatedon MTL-B. Single task learning baseline (STL-B) is used for MTL performance evaluation ∆ m .
As multi-gate MoE shows better performance than task-conditioned MoE, the reported M 2 ViT and
M 3 ViT results are based on the multi-gate design.
Results on PASCAL-Context Dataset As shown in Table 1, even using a vanilla MTL-B framework, introducing MoE (M 2 ViT) can achieve the highest performance over all previous encoderfocused works (+2.71% MTL performance); meanwhile, it significantly reduces their single taskinference FLOPs (particularly, reducing Cross-Stitch by 88%).
Comparing against Uncertainty,
DWA, GradNorm, and MGDA, the superior performance of M 2 ViT demonstrates its strong capacityin handling training conflict.
Moreover, leveraging the Model-Accelerator co-design helps us toconsume less than one-tenth the energy cost when deploying our model on FPGA.
Some qualitativeresults are shown in Figure 4.
CR indicates usage of our memory-efficient computation reordering method (Section 3.2)
[Table]
Results on NYUD-v2 Dataset On this dataset, M 2 ViT can reduce previous SoTA’s inference
FLOPs by 68% while achieving comparable MTL performance.
Introducing MoE to M-ViT helps toenlarge the model capacity without increasing inference FLOPs, which results in a MTL performanceboost from −6.27% to +1.59%.
With our Model-Accelerator co-design, we also see a nearly tenfoldincrease in energy efficiency.
Results are shown in Table
[Table]
4.3 Effect of Task-dependent MoE Design
To evaluate the effectiveness of our task-dependent MoE design, we compare between several modelsin Table 3 including STL-B, MTL ViT (M-ViT), M 3 ViT with one gating function for all the tasks,
M 3 ViT with multi gates
[Table]
and M 3 ViT with task-conditioned token input.
Results on both PASCALContext and NYUD-v2 datasets show that adding MoE layers into ViT with only one gating functionfor all tasks can already improve the M-ViT model.
Making MoE selection task-dependent canfurther improve the performance, where multi-gating performs better than task-conditioned gatingdesign.
Particularly, comparing our model performance with STL-B as well as previous SoTAs in
Table 1 and 2, we find that our MoE model better demonstrates its effectiveness when more tasksneed to be encapsulated in the system.
More results about our model’s performance on differentnumbers of tasks can be found in the supplement.
4.4 Hardware Performance Results
Results comparing hardware performance metrics on FPGA and GPU are shown in Table 4.
We firstdiscuss our memory efficiency.
The naive approach described in Section 3.2 would require 11.
MiB of on-chip memory (too much for our FPGA platform), but our compute-reordering designachieves the same result using only 4.840 MiB, demonstrating a 2.40 × reduction.
Moreover, whencomparing against a memory-constrained MoE ViT on FPGA without our compute-reordering method(using the cache-based method described in Section 3.2), we see that our method takes 9.23 × lesslatency and energy on the PASCAL-Context dataset and 8.88 × less latency and energy on NYUD-v2.
Our compute-reordering M 3 ViT on FPGA also beats all GPU baselines in energy efficiency; e.g., itbeats MoE ViT on GPU by 10.79 × on PASCAL-Context and by 9.69 × on NYUD-v2.
For discussionon the latency breakdown of M 3 ViT, please refer to the supplement.
Conclusion, Discussion of Limitation and Broader Impact
In this paper, we propose a model-accelerator co-design for efficient on-device MTL.
By customizing
MTL mixture-of-experts layers into a ViT backbone, we achieve the following: (i) we sparselyactivate task-specific experts in training to mitigate MTL gradient conflicts.
(ii) Given any task ofinterest, we can activate only the task-corresponding sparse “expert” pathway for efficiency.
Withour hardware-level co-design, we can achieve zero-overhead switching between tasks and can scaleto any number of experts.
Extensive experiments that show the proposed M 3 ViT surpasses thetop-performing encoder-focused MTL methods, reduces 88% FLOPs, and saves more than 8 × energyover our baseline.
The limitation of our work is that the proposed M 3 ViT is mainly evaluated on academic datasets;we will try real applications like autonomous driving in the future.
For broader impact, this workreduces the resource and energy consumption needed for MTL regimes, while still maintaining SoTAperformance, which can effectively serve the goal of GreenAI.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] please see section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] please seesection
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
Our work
does not contain theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A]
Our work does not
contain theoretical results.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] We used publicly available data in all of our experiments.
Meanwhile we either provide the detailed implementations or cite the papers of them following the authors instructions
(please refer to our implementation details and appendix).
All of our codes will be released.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes] We provides all the training details in appendix.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] We test our model on PASCAL-Context and NYUD-v for three times and report their mean results.
(d) Did you include the total amount of compute and the type of resources used (e.g.,
type of GPUs, internal cluster, or cloud provider)?
[Yes] We describe the details of
computation resources in the appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We used publicly
available data, i.e., PASCAL-Context, in our experiments.
We cited the corresponding
papers published by the creators.
(b) Did you mention the license of the assets?
[N/A]
The license of ImageNet, PASCAL-
Context, NYUD-v2 are included in the paper that we have cited.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
The ImageNet we used are publicly available.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
We did not collect/curate new data.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] ImageNet and the used MTL datasets are
already publicly available and broadly adopted.
We do not think there are any issues of
personally identifiable information or offensive content
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]