Original: A Tree-Structured Multi-Task Model Recommender
Revised: A Tree-Structured Multi-Task Model Recommender

Original: Anonymous 1 Anonymous Institution

Original: Abstract Tree-structured multi-task architectures have been employed to jointly tackle multiple vision tasks in the context of multi-task learning (MTL).
Revised: Abstract Tree-structured multi-task architectures have been employed to jointly tackle multiple vision tasks in the context of multi-task learning (MTL).

Original: The major challenge is to determine where to branch out for each task given a backbone model to optimize for both task accuracy and computation efficiency.
Revised: The major challenge is to determine where to branch out for each task given a backbone model to optimize for both task accuracy and computation efficiency.

Original: To address the challenge, this paper proposes a recommender that, given a set of tasks and a convolutional neural network-based backbone model, auto- matically suggests tree-structured multi-task architectures that could achieve a high task performance while meeting a user-specified computation budget without performing model training.
Revised: To address the challenge, this paper proposes a recommender that, given a set of tasks and a convolutional neural network-based backbone model, automatically suggests tree-structured multi-task architectures that could achieve a high task performance while meeting a user-specified computation budget without performing model training.

Original: Extensive evaluations on popular MTL benchmarks show that the recommended architectures could achieve competitive task accuracy and computation efficiency compared with state-of-the-art MTL methods.
Revised: Extensive evaluations on popular MTL benchmarks show that the recommended architectures could achieve competitive task accuracy and computation efficiency compared with state-of-the-art MTL methods.

Revised: Our tree-structured multi-task model recommender is open-sourced and available at https://github.com/zhanglijun95/TreeMTL .

Original: 1 Introduction
Revised: 1 Introduction

Original: Multi-task learning (MTL) aims to solve multiple tasks simultaneously.
Revised: Multi-task learning (MTL) aims to solve multiple tasks simultaneously.

Original: Compared to independently learning tasks, it is an effective approach to improve task performance while reducing computation and storage costs.
Revised: Compared to independently learning tasks, it is an effective approach to improve task performance while reducing computation and storage costs.

Original: However, over-sharing information between tasks can cause task interference (Sener and Koltun, 2018; Maninis et al., 2019) and accuracy degradation.
Revised: However, over-sharing information between tasks can cause task interference (Sener and Koltun, 2018; Maninis et al., 2019) and accuracy degradation.

Original: The major challenge in designing a multi-task architecture is thus to identify an intermediate state between over-shared and independent architectures (i.e., a partially-shared architecture), which not only preserves the benefits of lower computation cost and memory overhead, but also avoid task interference as much as possible to guarantee acceptable task accuracy.
Revised: The major challenge in designing a multi-task architecture is thus to identify an intermediate state between over-shared and independent architectures (i e., a partially-shared architecture), which not only preserves the benefits of lower computation cost and memory overhead, but also avoid task interference as much as possible to guarantee acceptable task accuracy.

Original: Such a partially-shared architecture is also called a tree-structured multi-task architecture .
Revised: Such a partially-shared architecture is also called a tree-structured multi-task architecture .

Original: Its shallow network layers are shared across tasks like tree roots, whereas deeper ones gradually grow more task-specific like tree branches (Vandenhende et al., 2019).
Revised: Its shallow network layers are shared across tasks like tree roots whereas deeper ones gradually grow more task-specific like tree branches (Vandenhende et al., 2019).

Original: Identifying the best tree-structured multi-task architecture needs to determine where to branch out for each task to optimize for both computation efficiency and task accuracy.
Revised: Identifying the best tree-structured multi-task architecture needs to determine where to branch out for each task to optimize for both computation efficiency and task accuracy.

Original: Previous works opted for the simplest strategy of sharing the initial layers of a backbone model, after which all tasks branch out simultaneously (Ruder, 2017; Nekrasov et al., 2019; Suteu and Guo, 2019; Leang et al., 2020).
Revised: Previous works opted for the simplest strategy of sharing the initial layers of a backbone model, after which all tasks branch out simultaneously (Ruder, 2017; Nekrasov et al., 2019; Suteu and Guo, 2019; Leang et al., 2020).

Original: Since the point at which the branching occurs is determined manually, they call for domain expertise when tackling different tasks and usually result in unsatisfactory solutions due to the enormous architecture design space.
Revised: Since the point at which the branching occurs is determined manually, they call for domain expertise when tackling different tasks and usually result in unsatisfactory solutions due to the enormous architecture design space.

Original: To automate architecture design, one line of work deduced the layer sharing possibility based on measurable task relatedness (Lu et al., 2017;
Revised: To automate architecture design, one line of work deduced the layer sharing possibility based on measurable task relatedness (Lu et al., 2017;

Original: Vandenhende et al., 2019; Standley et al., 2020) and minimized the total task dissimilarity when designing multi-task architectures.
Revised: Vandenhende et al., 2019; Standley et al., 2020) and minimized the total task dissimilarity when designing multi-task architectures.

Original: However, they ignore task interactions that could bring the potential generalization improvement and positive inhibition of overfitting when multiple tasks are trained together (Ruder, 2017; Vandenhende et al., 2020).
Revised: However, they ignore task interactions that could bring the potential generalization improvement and positive inhibition of overfitting when multiple tasks are trained together (Ruder, 2017; Vandenhende et al., 2020).

Original: Another line of work attempted to learn how to branch a network such that the overall multi-task loss is minimized via differentiable neural architecture search (Bruggemann et al., 2020; Guo et al., 2020).
Revised: Another line of work attempted to learn how to branch a network such that the overall multi-task loss is minimized via differentiable neural architecture search (Bruggemann et al., 2020; Guo et al., 2020).

Original: Such end-to-end frameworks integrated the architecture search with the network training process, which easily leads to sub-optimal multi-task architectures (Choromanska et al., 2015; Sun et al., 2020) due to training difficulties.
Revised: Such end-to-end frameworks integrated the architecture search with the network training process, which easily leads to sub-optimal multi-task architectures (Choromanska et al., 2015; Sun et al., 2020) due to training difficulties.

Original: Besides, the learned multi-task architectures cannot guarantee to meet a user-defined computation budget since these methods are like a black box where users cannot control the exploring process.
Revised: Besides, the learned multi-task architectures cannot guarantee to meet a user-defined computation budget since these methods are like a black box where users cannot control the exploring process.

Original: In this paper, we overcome the aforementioned limitations and propose a tree-structured multitask model recommender.
Revised: In this paper, we overcome the aforementioned limitations and propose a tree-structured multitask model recommender.

Original: It takes as inputs an arbitrary convolutional neural network (CNN)© 2022 the authors, BY 4. backbone model and a set of tasks in interest, and then predicts the top- 𝑘 tree-structured multi-task architectures that achieve high task accuracy while meeting a user-specified computation budget.
Revised: It takes as inputs an arbitrary convolutional neural network (CNN) backbone model and a set of tasks in interest, and then predicts the top- 𝑘 tree-structured multi-task architectures that achieve high task accuracy while meeting a user-specified computation budget.

Original: Our basic idea is to build a task accuracy estimator that can predict the task accuracy of each multi-task model architecture in the design space without performing model training.
Revised: Our basic idea is to build a task accuracy estimator that can predict the task accuracy of each multi-task model architecture in the design space without performing model training.

Original: The task accuracy estimator captures task interactions by leveraging the task performance of well-trained two-task architectures instead and enables ranking of all multi-task architectures with more than two tasks using their predicted task accuracy.
Revised: The task accuracy estimator captures task interactions by leveraging the task performance of well-trained two-task architectures instead and enables ranking of all multi-task architectures with more than two tasks using their predicted task accuracy.

Original: The recommender can then enumerate the design space and identify the multi-task models with the highest predicted task accuracy.
Revised: The recommender can then enumerate the design space and identify the multi-task models with the highest predicted task accuracy.

Original: Unlike differentiable neural architecture search-based approaches, the recommender is a white-box that allows users to easily control the computation complexity of the multi-task architectures.
Revised: Unlike differentiable neural architecture search-based approaches, the recommender is a white-box that allows users to easily control the computation complexity of the multi-task architectures.

Original: The basic idea poses three major research questions:
Revised: The basic idea poses three major research questions:

Original: • RQ 1: how to build an accurate task accuracy estimator that enables a faithful ranking of the multi-task architectures in the design space based on their estimated task performance?
Revised: • RQ 1: how to build an accurate task accuracy estimator that enables a faithful ranking of the multi-task architectures in the design space based on their estimated task performance?

Original: • RQ 2: how to represent multi-task model architectures such that a recommender can completely enumerate the design space for estimating their performance?
Revised: • RQ 2: how to represent multi-task model architectures such that a recommender can completely enumerate the design space for estimating their performance?

Original: • RQ 3: how to automatically support various CNN backbone models?
Revised: • RQ 3: how to automatically support various CNN backbone models?

Original: To answer RQ 1 , our task accuracy estimator predicts the task accuracy of a multi-task architecture by averaging the task accuracy of associated well-trained two-task architectures.
Revised: To answer RQ 1 , our task accuracy estimator predicts the task accuracy of a multi-task architecture by averaging the task accuracy of associated well-trained two-task architectures.

Original: A ranking score of the multi-task architecture is calculated as the weighted sum of the tasks’ accuracy, where the weight of each task is determined by quantified accuracy variance to ensure faithful ranking.
Revised: A ranking score of the multi-task architecture is calculated as the weighted sum of the tasks’ accuracy, where the weight of each task is determined by quantified accuracy variance to ensure faithful ranking.

Original: To answer RQ 2 , we propose a novel data structure called Layout to represent a multi-task architecture and an operation called Layout Cut to derive multi-task architectures.
Revised: To answer RQ 2 , we propose a novel data structure called Layout to represent a multi-task architecture and an operation called Layout Cut to derive multi-task architectures.

Original: We further propose a cut-based recursive algorithm that is proved to be able to enumerate the design space completely.
Revised: We further propose a cut-based recursive algorithm that is proved to be able to enumerate the design space completely.

Original: To answer RQ 3 , we design a branching point detector to automatically separate a CNN backbone model into a sequence of computation blocks where each block corresponds to a possible branching point.
Revised: To answer RQ 3 , we design a branching point detector to automatically separate a CNN backbone model into a sequence of computation blocks where each block corresponds to a possible branching point.

Original: 1 The detector saves manual efforts in applying the recommender to an arbitrary
Revised: 1 The detector saves manual efforts in applying the recommender to an arbitrary

Original: CNN architecture.
Revised: CNN architecture.

Original: Experiments on popular MTL benchmarks, NYUv2 (Silberman et al., 2012) and Tiny-Taskonomy (Zamir et al., 2018), using different backbone models, Deeplab-ResNet34 (Chen et al., 2017) and
Revised: Experiments on popular MTL benchmarks, NYUv2 (Silberman et al., 2012) and Tiny-Taskonomy (Zamir et al., 2018), using different backbone models, Deeplab-ResNet34 (Chen et al., 2017) and

Original: MobileNetV2 (Sandler et al., 2018), demonstrate that the recommended tree-structured multi-task architectures achieve competitive task accuracy compared with state-of-the-art MTL methods under specified computation budgets.
Revised: MobileNetV2 (Sandler et al., 2018), demonstrate that the recommended tree-structured multi-task architectures achieve competitive task accuracy compared with state-of-the-art MTL methods under specified computation budgets.

Original: Our empirical evaluation also demonstrates that ranking of the multi-task architectures using estimated task accuracy without training has a high correlation (Pearson’s 𝛾 is 0 .
Revised: Our empirical evaluation also demonstrates that ranking of the multi-task architectures using estimated task accuracy without training has a high correlation (Pearson’s 𝛾 is 0 .

Original: 5 ∼ 0 . 85) with the oracle ranking after training for different CNN architectures.
Revised: 5 ∼ 0 . 85) with the oracle ranking after training for different CNN architectures.

Original: 2 Related Works
Revised: 2 Related Works

Original: Multi-task learning (MTL) is commonly categorized into either hard or soft parameter sharing (Ruder, 2017; Vandenhende et al., 2020).
Revised: Multi-task learning (MTL) is commonly categorized into either hard or soft parameter sharing (Ruder, 2017; Vandenhende et al., 2020).

Original: In hard parameter sharing, a set of parameters in the backbone model are shared among tasks.
Revised: In hard parameter sharing, a set of parameters in the backbone model are shared among tasks.

Original: In soft parameter sharing (Misra et al., 2016; Ruder et al., 2019; Gao et al., 2019), each task has its own set of parameters.
Revised: In soft parameter sharing (Misra et al., 2016; Ruder et al., 2019; Gao et al., 2019), each task has its own set of parameters.

Original: Task information is shared by applying regularization on parameters during training, such as enforcing the weights of the model for each task to be similar.
Revised: Task information is shared by applying regularization on parameters during training, such as enforcing the weights of the model for each task to be similar.

Original: In this paper, we focus on hard parameter sharing as it produces memoryand computation-efficient multi-task models.
Revised: In this paper, we focus on hard parameter sharing as it produces memoryand computation-efficient multi-task models.

Original: Early works on multi-task architecture design rely on domain expertise to decide which layers should be shared across tasks and which ones should be task-specific (Long et al., 2017; Nekrasov
Revised: Branching Point Detector

Original: [Figure]
Revised: [Figure]

Revised: model

Revised: Model with branching points

Original: Design Space Enumerator
Revised: Design Space Enumerator

Original: Multi [Figure] task [Figure]
Revised: Multi-task model candidates

Original: candidates

Original: Task Accuracy Estimator 3 Tasks
Revised: Task Accuracy Estimator 3 Tasks

Original: Performance table
Revised: Performance table

Original: Model
Revised: Model

Original: Accuracy ↓
Revised: Accuracy ↓

Original: Flops # 87.
Revised: Flops # 87.

Original: > C # 87.
Revised: > C # 87.

Original: < C # 86. < C # 86.
Revised: < C # 86. < C # 86.

Original: > C … … …
Revised: > C … … …

Original: Computational Budget C
Revised: Computational Budget C

Original: Available  et al., 2019; Suteu and Guo, 2019; Leang et al., 2020).
Revised: Available Early works on multi-task architecture design rely on domain expertise to decide which layers should be shared across tasks and which ones should be task-specific (Long et al., 2017; Nekrasov et al., 2019; Suteu and Guo, 2019; Leang et al., 2020).

Original: Due to the enormous design space, such approaches are difficult to find an optimal solution.
Revised: Due to the enormous design space, such approaches are difficult to find an optimal solution.

Original: In recent years, researchers attempt to automate the procedure of designing multi-task architectures.
Revised: In recent years, researchers attempt to automate the procedure of designing multi-task architectures.

Original: Deep Elastic Network (DEN) (Ahn et al., 2019) uses reinforcement learning (RL) to determine whether each filter in convolutional layers can be shared across tasks.
Revised: Deep Elastic Network (DEN) (Ahn et al., 2019) uses reinforcement learning (RL) to determine whether each filter in convolutional layers can be shared across tasks.

Original: Similarly, AdaShare (Sun et al., 2019) and AutoMTL (Zhang et al., 2021) learn task-specific policies that select which layers to execute for a given task.
Revised: Similarly, AdaShare (Sun et al., 2019) and AutoMTL (Zhang et al., 2021) learn task-specific policies that select which layers to execute for a given task.

Original: Some other works (Gao et al., 2020;
Revised: Some other works (Gao et al., 2020;

Original: Wu et al., 2021) adopt NAS techniques to explore feature fusion opportunities across tasks.
Revised: Wu et al., 2021) adopt NAS techniques to explore feature fusion opportunities across tasks.

Original: Their primary goal is to improve task accuracy instead of computation efficiency by minimizing the overall multi-task loss.
Revised: Their primary goal is to improve task accuracy instead of computation efficiency by minimizing the overall multi-task loss.

Original: Thus there is no guarantee that the searched multi-task model architectures will meet the computation budget.
Revised: Thus there is no guarantee that the searched multi-task model architectures will meet the computation budget.

Original: Also, their architecture search procedure requires substantial search time and is usually hard to converge since the sharing strategy and network parameters generally prefer the alternating training principle to stabilize the training process (Xie et al., 2018; Sun et al., 2019; Wu et al., 2019).
Revised: Also, their architecture search procedure requires substantial search time and is usually hard to converge since the sharing strategy and network parameters generally prefer the alternating training principle to stabilize the training process (Xie et al., 2018; Sun et al., 2019; Wu et al., 2019).

Original: Our work pays more attention to balancing task accuracy and computation efficiency through recommending branching structures for multi-task models.
Revised: Our work pays more attention to balancing task accuracy and computation efficiency through recommending branching structures for multi-task models.

Original: There also exist several interesting methods under this direction.
Revised: There also exist several interesting methods under this direction.

Original: FAFS (Lu et al., 2017) starts from a thin network where tasks initially share all layers and dynamically grows the model in a greedy layer-by-layer fashion depending on task similarities.
Revised: FAFS (Lu et al., 2017) starts from a thin network where tasks initially share all layers and dynamically grows the model in a greedy layer-by-layer fashion depending on task similarities.

Original: It computes task similarity based on the likelihood of input samples having the same difficulty level.
Revised: It computes task similarity based on the likelihood of input samples having the same difficulty level.

Original: What-to-Share (Vandenhende et al., 2019) measures the task affinity by analyzing the representation similarity between independent models for each task.
Revised: What-to-Share (Vandenhende et al., 2019) measures the task affinity by analyzing the representation similarity between independent models for each task.

Original: It recommends the multi-task architecture with the minimum total task dissimilarity.
Revised: It recommends the multi-task architecture with the minimum total task dissimilarity.

Original: However, because the task dissimilarity between two tasks is always non-negative, the theoretical optimal multi-task architecture would be always independent models whose total task dissimilarity is zero.
Revised: However, because the task dissimilarity between two tasks is always non-negative, the theoretical optimal multi-task architecture would be always independent models whose total task dissimilarity is zero.

Original: In contrast to pre-computing the task relatedness, BMTAS (Bruggemann et al., 2020) and Learn-to-Branch (Guo et al., 2020) utilize differentiable neural architecture search to construct end-to-end trainable frameworks that integrate the architecture exploration with the network training process.
Revised: In contrast to pre-computing the task relatedness, BMTAS (Bruggemann et al., 2020) and Learn-to-Branch (Guo et al., 2020) utilize differentiable neural architecture search to construct end-to-end trainable frameworks that integrate the architecture exploration with the network training process.

Original: These learning-based methods easily lead to the suboptimal multi-task model (Choromanska et al., 2015; Sun et al., 2020) due to difficulties in training and cannot guarantee the resulting multi-task architecture to obey a user-defined computation budget.
Revised: These learning-based methods easily lead to the suboptimal multi-task model (Choromanska et al., 2015; Sun et al., 2020) due to difficulties in training and cannot guarantee the resulting multi-task architecture to obey a user-defined computation budget.

Original: 3 Proposed Approach
Revised: 3 Proposed Approach

Original: Given a backbone model with 𝐵 branching point and a set of 𝑇 tasks, our goal is to build a recommender that, when deployed, predicts 𝑘 tree-structured multi-task architectures that achieve a high task accuracy while meeting a user-specified computation budget 𝐶 .
Revised: Given a backbone model with 𝐵 branching point and a set of 𝑇 tasks, our goal is to build a recommender that, when deployed, predicts 𝑘 tree-structured multi-task architectures that achieve a high task accuracy while meeting a user-specified computation budget 𝐶 .

Original: Figure 1 illustratesthe offline building process and the online usage of the recommender. During the offline building process, users provide an arbitrary CNN-based backbone model and a set of tasks. A branching point
Revised: Figure 1 illustrates

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Revised: the offline building process and the online usage of the recommender.

Revised: During the offline building process, users provide an arbitrary CNN-based backbone model and a set of tasks.

Original:  detector will automatically identify the sequential computational blocks in the backbone model,[Figure] each of the blocks corresponding to a viable branching point.
Revised: A branching point detector will automatically identify the sequential computational blocks in the backbone model, each of the blocks corresponding to a viable branching point.

Original: A task accuracy estimator is then built based on the given set of tasks and the identified branching points to predict the performance of all the tree-structured multi-task architectures in the design space, which are explored using a design[Figure] space enumerator .
Revised: A task accuracy estimator is then built based on the given set of tasks and the identified branching points to predict the performance of all the tree-structured multi-task architectures in the design space, which are explored using a design space enumerator .

Original: The performance of these multi-task architectures including their other attributes such as model size, FLOPs etc. could be stored in a performance table to facilitate online queries.
Revised: The performance of these multi-task architectures including their other attributes such as model size, FLOPs etc. could be stored in a performance table to facilitate online queries.

Original: When deployed, the recommender takes a user-specified computation budget 𝐶 as input, and suggests multi-task architectures by looking up the performance table on the fly.
Revised: When deployed, the recommender takes a user-specified computation budget 𝐶 as input, and suggests multi-task architectures by looking up the performance table on the fly.

Original: We next elaborate the three major components, task accuracy estimator , design space enumerator , and branching point detector in detail.
Revised: We next elaborate the three major components, task accuracy estimator , design space enumerator , and branching point detector in detail.

Original: 3.1 Task Accuracy Estimator
Revised: 3.1 Task Accuracy Estimator

Original: Task accuracy estimator predicts the task accuracy of a tree-structured multi-task architecture without performing actual model training.
Revised: Task accuracy estimator predicts the task accuracy of a tree-structured multi-task architecture without performing actual model training.

Original: The problem is challenging because predicting a single-task architecture’s accuracy is already non-trivial and multi-task architectures introduce more complexities due to task interactions and interference.
Revised: The problem is challenging because predicting a single-task architecture’s accuracy is already non-trivial and multi-task architectures introduce more complexities due to task interactions and interference.

Original: Task accuracy estimator addresses the problem by leveraging well-trained two-task architectures to quantify task accuracy and interactions and predict the performance of a multi-task architecture.
Revised: Task accuracy estimator addresses the problem by leveraging well-trained two-task architectures to quantify task accuracy and interactions and predict the performance of a multi-task architecture.

Original: Specifically, for any tree-structured multitask architecture, the estimator predicts its task accuracy by averaging the task accuracy of all the associated two-task architectures.
Revised: Specifically, for any tree-structured multitask architecture, the estimator predicts its task accuracy by averaging the task accuracy of all the associated two-task architectures.

Original: A two-task architecture is considered associated if it meets both conditions: (1) the two tasks are a subset of the tasks in the multi-task architecture; (2) the two tasks have the identical branching point as in the multi-task architecture.
Revised: A two-task architecture is considered associated if it meets both conditions: (1) the two tasks are a subset of the tasks in the multi-task architecture; (2) the two tasks have the identical branching point as in the multi-task architecture.

Original: The algorithm pseudocode for identifying associated two-task architectures is in the Appendix Section ?? .

Original: Figure 2 illustrate the basic idea.
Revised: Figure 2 illustrate the basic idea.

Original: Figure 2(a) shows a multi-task architecture constructed from a backbone model with five branching points for three tasks and Figure 2(b) shows the three associated two-task architectures.
Revised: Figure 2(a) shows a multi-task architecture constructed from a backbone model with five branching points for three tasks and Figure 2(b) shows the three associated two-task architectures.

Original: The numbers inside each block indicate among which tasks the block is shared.
Revised: The numbers inside each block indicate among which tasks the block is shared.

Original: Tasks 1 and 2 branch out after the third block, which is the same branching point as the first two-task architecture in Figure 2(b).
Revised: Tasks 1 and 2 branch out after the third block, which is the same branching point as the first two-task architecture in Figure 2(b).

Original: Similarly, tasks 1 and 3 branch out after the second block, which is the same branching point as the second two-task architecture in Figure 2(b).
Revised: Similarly, tasks 1 and 3 branch out after the second block, which is the same branching point as the second two-task architecture in Figure 2(b).

Original: We estimate task 1 accuracy of the multi-task architecture by averaging task 1 accuracy of the first and second two-task architectures, task 2 accuracy from those of the first and third two-task models, and task 3 accuracy from those of the second and third two-task models.
Revised: We estimate task 1 accuracy of the multi-task architecture by averaging task 1 accuracy of the first and second two-task architectures, task 2 accuracy from those of the first and third two-task models, and task 3 accuracy from those of the second and third two-task models.

Original: The ultimate goal of the task accuracy estimator is to enable ranking of multi-task architectures based on their estimated task accuracy.
Revised: The ultimate goal of the task accuracy estimator is to enable ranking of multi-task architectures based on their estimated task accuracy.

Original: Due to the noise in training two-task architectures (Pham et al., 2020), an estimated task accuracy of a multi-task architecture could suffer from some accuracy variance and lead to an inaccurate ranking.
Revised: Due to the noise in training two-task architectures (Pham et al., 2020), an estimated task accuracy of a multi-task architecture could suffer from some accuracy variance and lead to an inaccurate ranking.

Original: A ranking score is thus calculated as the weighted sum of the tasks’ performance.
Revised: A ranking score is thus calculated as the weighted sum of the tasks’ performance.

Original: Tasks with higher accuracy variance have lower task weight .
Revised: Tasks with higher accuracy variance have lower task weight .

Original: To quantify accuracy variance and task weight, we adopt the Singular Value Decomposition
Revised: To quantify accuracy variance and task weight, we adopt the Singular Value Decomposition

Original: Entropy (SVDE) (Li et al., 2008; Jelinek et al., 2019) to measure the regularity of each task 𝑡 𝑖 ’s performance in its 𝐵 + 1 two-task architectures with another task 𝑡 𝑗 .
Revised: Entropy (SVDE) (Li et al., 2008; Jelinek et al., 2019) to measure the regularity of each task 𝑡 𝑖 ’s performance in its 𝐵 + 1 two-task architectures with another task 𝑡 𝑗 .

Revised: SVDE has two important features that make it suitable for regularity measurement.

Revised: (1) SVDE is scale-invariant — that is, it characterizes the accuracy trend rather than the absolute values of the accuracy sequence.

Revised: SVDE is order-aware since it is designed for data sequences, not just for a set of data.

Revised: The order of data items affects data regularity, which is captured by SVDE.

Original: SVDE reflects the number of orthogonal vectors contributed to a task performance sequence ( Δ 𝑡 ( 0 ) 𝑖 , . . .
Revised: SVDE reflects the number of orthogonal vectors contributed to a task performance sequence ( Δ 𝑡 ( 0 ) 𝑖 , . . .

Original: , Δ 𝑡 ( 𝐵 ) 𝑖 | 𝑡 𝑖 , 𝑡 𝑗 ) , where Δ 𝑡 ( 𝑏 ) 𝑖 is 𝑡 𝑖 ’s performance in a two-task model that branches at 𝑏 -th point.
Revised: , Δ 𝑡 ( 𝐵 ) 𝑖 | 𝑡 𝑖 , 𝑡 𝑗 ) , where Δ 𝑡 ( 𝑏 ) 𝑖 is 𝑡 𝑖 ’s performance in a two-task model that branches at 𝑏 -th point.

Original: Higher entropy indicates lower regularity and thus higher variance.
Revised: Higher entropy indicates lower regularity and thus higher variance.

Original: The task weight of 𝑡 𝑖 is the average of the negative entropy over all possible two task combinations ( 𝑡 𝑖, 𝑡 𝑗 ) , ∀ 𝑗 ≠ 𝑖 : 𝑤
Revised: The task weight of 𝑡 𝑖 is the average of the negative entropy over all possible two task combinations ( 𝑡 𝑖

Original: [Equation]
Revised: [Equation]

Original: |
Revised: 𝑡 𝑗 ) ,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) where T is the set of tasks and 𝑇 =
Revised: where T is the set of tasks and 𝑇 =

Original: | T | is the number of tasks.
Revised: | T | is the number of tasks.

Original: The final ranking score of a multi-task architecture is:
Revised: The final ranking score of a multi-task architecture is:

Original: [Equation]
Revised: [Equation]

Original: where 𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒 ([ 𝑤 1 , . . . ,𝑤 𝑇 ]) is the normalized task weights so that their sum is equal to one.
Revised: where 𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒 ([ 𝑤 1 , . . . ,𝑤 𝑇 ]) is the normalized task weights so that their sum is equal to one.

Original: Estimating the accuracy of a multi-task architecture requires training of its associated two-task architectures.
Revised: Estimating the accuracy of a multi-task architecture requires training of its associated two-task architectures.

Original: Given 𝑇 number of tasks and 𝐵 number of branching points, the total number of two-task architectures to train is 𝐶 2 𝑇 · ( 𝐵 + 1 ) , where 𝐶 2 𝑇 is the number of two task combinations.
Revised: Given 𝑇 number of tasks and 𝐵 number of branching points, the total number of two-task architectures to train is 𝐶 2 𝑇 · ( 𝐵 + 1 ) , where 𝐶 2 𝑇 is the number of two task combinations.

Original: The training overhead of the two-task architectures is much less than training all the multi-task architectures whose number is 𝑂 ([( 2 𝑇 − 1 − 1 ) · 𝐵 ] 𝑇 − 1 ) .
Revised: The training overhead of the two-task architectures is much less than training all the multi-task architectures whose number is 𝑂 ([( 2 𝑇 − 1 − 1 ) · 𝐵 ] 𝑇 − 1 ) .

Original: Our experiments in Section 4.3 demonstrate that the ranking of multi-task architectures using estimated task accuracy without training has a high correlation (Pearson’s 𝛾 is 0 .
Revised: Our experiments in Section 4.3 demonstrate that the ranking of multi-task architectures using estimated task accuracy without training has a high correlation (Pearson’s 𝛾 is 0 .

Original: 5 ∼ 0 . 85) with the oracle ranking from training for different CNN architectures.
Revised: 5 ∼ 0 . 85) with the oracle ranking from training for different CNN architectures.

Original: 3.2 Design Space Enumerator
Revised: 3.2 Design Space Enumerator

Original: Design space enumerator formalizes the representation of tree-structured multi-task architectures so that the recommender can completely enumerate the design space.
Revised: Design space enumerator formalizes the representation of tree-structured multi-task architectures so that the recommender can completely enumerate the design space.

Original: It introduces a data structure called Layout and an operator called Layout Cut to derive multi-task architectures.
Revised: It introduces a data structure called Layout and an operator called Layout Cut to derive multi-task architectures.

Original: Based on the two core concepts, we propose a cut-based recursive algorithm to enumerate all possible architectures.
Revised: Based on the two core concepts, we propose a cut-based recursive algorithm to enumerate all possible architectures.

Original: Definition 3.1 (Layout) .
Revised: Definition 3.1 (Layout) .

Original: A layout is a symbolized representation of a tree-structured multi-task architecture.
Revised: A layout is a symbolized representation of a tree-structured multi-task architecture.

Original: Formally, for 𝑇 tasks and a backbone model with 𝐵 branching points, a layout L =
Revised: Formally, for 𝑇 tasks and a backbone model with 𝐵 branching points, a layout L =

Original: [ 𝐿 1 , 𝐿 2 , · · · , 𝐿 𝐵 ] , where 𝐿 𝑖 is a list of task sets at the 𝑖 -th branching point.
Revised: [ 𝐿 1 , 𝐿 2 , · · · , 𝐿 𝐵 ] , where 𝐿 𝑖 is a list of task sets at the 𝑖 -th branching point.

Original: Task sets in 𝐿 𝑖 =
Revised: Task sets in 𝐿 𝑖 =

Original: [ 𝐿 1 𝑖 , 𝐿 2 𝑖 , · · · ] are subsets of tasks T and satisfy two conditions: (1) 𝐿 1 𝑖 ∪ 𝐿 2 𝑖 ∪· · · = T , and (2) 𝐿 𝑝𝑖 ∩ 𝐿 𝑞𝑖 = ∅ , ∀ 𝐿 { 𝑝,𝑞 } 𝑖 ∈ 𝐿 𝑖 .
Revised: [ 𝐿 1 𝑖 , 𝐿 2 𝑖 , · · · ] are subsets of tasks T and satisfy two conditions: (1) 𝐿 1 𝑖 ∪ 𝐿 2 𝑖 ∪· · · = T , and (2) 𝐿 𝑝𝑖 ∩ 𝐿 𝑞𝑖 = ∅ , ∀ 𝐿 { 𝑝,𝑞 } 𝑖 ∈ 𝐿 𝑖 .

Original: A task set 𝐿 𝑝𝑖 means the set of tasks in 𝐿 𝑝𝑖 sharing the 𝑖 -th block.
Revised: A task set 𝐿 𝑝𝑖 means the set of tasks in 𝐿 𝑝𝑖 sharing the 𝑖 -th block.

Original: Figure 2 illustrates the layouts of a multi-task architecture and three two-task architectures.
Revised: Figure 2 illustrates the layouts of a multi-task architecture and three two-task architectures.

Original: We define the initial layout as
Revised: We define the initial layout as

Original: L 0 =
Revised: L 0 =

Original: [[ T ] , · · · , [ T ]
Revised: [[ T ] , · · · , [ T ]

Original: (cid:124) (cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32) (cid:123)(cid:122) (cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32) (cid:125)

Original: 𝐵 ] =

Original: [Equation]
Revised: [Equation]

Original: [{ 𝑡 1 , . . . , 𝑡 𝑇 [Equation] [Equation] [{ 𝑡
Revised: (cid:123)(cid:122) (cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 𝑇
Revised: 𝐵

Original: [Equation]
Revised: [Equation]

Original: ] (cid:124) (cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)

Original: [Table]
Revised: [Table]

Original:  ] , which means all the tasks share all the blocks in the multi-task model.
Revised: 𝐵 ] , which means all the tasks share all the blocks in the multi-task model.

Original: Definition 3.2 (Layout Cut) .
Revised: Definition 3.2 (Layout Cut) .

Original: A layout cut is an operator that transforms one layout to another layout by selecting an available task set and dividing it into two task sets.
Revised: A layout cut is an operator that transforms one layout to another layout by selecting an available task set and dividing it into two task sets.

Original: (The complete definition can be found in the Appendix Section ?? )

Original: Based on Definition 3.2, we propose a cut-based algorithm to enumerate all possible layouts, namely tree-structured multi-task architectures.
Revised: Based on Definition 3.2, we propose a cut-based algorithm to enumerate all possible layouts, namely tree-structured multi-task architectures.

Original: The main idea is to recursively apply layout cuts on the initial layout L 0 and all the generated layouts until no new layout is generated.
Revised: The main idea is to recursively apply layout cuts on the initial layout L 0 and all the generated layouts until no new layout is generated.

Original: The pseudocode of the layouts enumerator is included in the Appendix Section ?? .

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: The cut-based layout enumeration algorithm could explore the design space of treestructured multi-task models completely.
Revised: The cut-based layout enumeration algorithm could explore the design space of treestructured multi-task models completely.

Original: This completeness theorem is proved by induction as demonstrated in the Appendix Section ?? .

Original: 3.3 Branching Point Detector
Revised: 3.3 Branching Point Detector

Original: Branching point detector allows the recommender to support an arbitrary CNN backbone model without manual reimplementation.
Revised: Branching point detector allows the recommender to support an arbitrary CNN backbone model without manual reimplementation.

Original: Its design is motivated by the observation that common CNN backbone models are a sequence of computation blocks , such as residual blocks in ResNet50 (He et al., 2016) and bottleneck blocks in MobileNetV2 (Sandler et al., 2018).
Revised: Its design is motivated by the observation that common CNN backbone models are a sequence of computation blocks , such as residual blocks in ResNet50 (He et al., 2016) and bottleneck blocks in MobileNetV2 (Sandler et al., 2018).

Original: These computation blocks are typically treated as branching points in MTL (Vandenhende et al., 2019; Bruggemann et al., 2020) and satisfy two requirements.
Revised: These computation blocks are typically treated as branching points in MTL (Vandenhende et al., 2019; Bruggemann et al., 2020) and satisfy two requirements.

Original: (1) They contain trainable parameters so that whether they are shared across tasks is likely to make a difference in task accuracy.
Revised: (1) They contain trainable parameters so that whether they are shared across tasks is likely to make a difference in task accuracy.

Original: (2) They are connected to each other sequentially–that is, there is no link across non-sequential blocks.
Revised: (2) They are connected to each other sequentially–that is, there is no link across non-sequential blocks.

Original: The two requirements inspire us to design a two-stage branching point detector.
Revised: The two requirements inspire us to design a two-stage branching point detector.

Original: The first stage is to identify groups of operators called candidate blocks in a given backbone model.
Revised: The first stage is to identify groups of operators called candidate blocks in a given backbone model.

Original: Each candidate block is a subgraph in the computation graph of the backbone model that takes only one input tensor and produces only one output tensor.
Revised: Each candidate block is a subgraph in the computation graph of the backbone model that takes only one input tensor and produces only one output tensor.

Original: The branching point detector leverages the Cut Theorem 2 in the Graph Theory to partition the original computation graph of the backbone model into candidate blocks.
Revised: The branching point detector leverages the Cut Theorem 2 in the Graph Theory to partition the original computation graph of the backbone model into candidate blocks.

Original: A subgraph can be divided into two subgraphs if the size of the minimum cut is one; otherwise, the subgraph can be no longer partitioned and is a candidate block.
Revised: A subgraph can be divided into two subgraphs if the size of the minimum cut is one; otherwise, the subgraph can be no longer partitioned and is a candidate block.

Original: Because a candidate block could contain operators that have no parameters at all, the second stage is to merge candidate blocks that contain only unparameterized layers (e.g., ReLU,
Revised: Because a candidate block could contain operators that have no parameters at all, the second stage is to merge candidate blocks that contain only unparameterized layers (e.g., ReLU,

Original: Pooling) and normalization layers (e.g., Batch Normalization) with adjacent candidate blocks (e.g.,
Revised: Pooling) and normalization layers (e.g., Batch Normalization) with adjacent candidate blocks (e.g.,

Original: Convolution Layer) to generate final computation blocks (e.g., ConvBNReLU).
Revised: Convolution Layer) to generate final computation blocks (e.g., ConvBNReLU).

Original: Each computation block corresponds to a viable branching point.
Revised: Each computation block corresponds to a viable branching point.

Original: The detailed algorithm and pseudocode are in the

Original: Appendix Section ?? .

Original: The proposed branching point detector enables the recommender to automatically parse the backbone model and produce two-task architectures and multi-task architectures based on a layout.
Revised: The proposed branching point detector enables the recommender to automatically parse the backbone model and produce two-task architectures and multi-task architectures based on a layout.

Original: It saves manual efforts in generalizing multi-task architecture search across different backbone models.
Revised: It saves manual efforts in generalizing multi-task architecture search across different backbone models.

Original: We also allow users to flexibly add or remove branching points to adjust the architecture search space.
Revised: We also allow users to flexibly add or remove branching points to adjust the architecture search space.

Original: 4 Experiments 
Revised: 4 Experiments 4.1 Experiment Settings

Original: 4.1 Experiment Settings

Original: Datasets and Tasks.
Revised: Datasets and Tasks.

Original: Our experiments are conducted on two popular datasets in multi-task learning,
Revised: Our experiments are conducted on two popular datasets in multi-task learning,

Original: NYUv2 (Silberman et al., 2012) and Tiny-Taskonomy (Zamir et al., 2018).
Revised: NYUv2 (Silberman et al., 2012) and Tiny-Taskonomy (Zamir et al., 2018).

Original: The NYUv2 dataset consists of RGB-D indoor scene images and three tasks, 13-class semantic segmentation, depth estimation, and surface normal prediction.
Revised: The NYUv2 dataset consists of RGB-D indoor scene images and three tasks, 13-class semantic segmentation, depth estimation, and surface normal prediction.

Original: Tiny-Taskonomy contains indoor images and five tasks: semantic segmentation, surface normal prediction, depth estimation, keypoint detection, and edge detection.
Revised: Tiny-Taskonomy contains indoor images and five tasks: semantic segmentation, surface normal prediction, depth estimation, keypoint detection, and edge detection.

Original: The data splits follow prior works (Sun et al., 2019; Zhang et al., 2021).
Revised: The data splits follow prior works (Sun et al., 2019; Zhang et al., 2021).

Original: Loss Functions and Evaluation Metrics.
Revised: Loss Functions and Evaluation Metrics.

Original: In NYUv2, Semantic segmentation uses a pixel-wise cross-entropy loss for each predicted class label, and is evaluated using mean Intersection over
Revised: In NYUv2, Semantic segmentation uses a pixel-wise cross-entropy loss for each predicted class label, and is evaluated using mean Intersection over

Original: Union and Pixel Accuracy (mIoU and Pixel Acc, the higher the better).
Revised: Union and Pixel Accuracy (mIoU and Pixel Acc, the higher the better).

Original: Surface normal prediction uses the inverse of cosine similarity between the normalized prediction and ground truth, and is evaluated using mean and median angle distances between the prediction and the ground truth (the lower the better), and the percentage of pixels whose prediction is within the angles of 11 . 25°, 22 . 5° and 30° to the ground truth (Eigen and Fergus, 2015) (the higher the better).
Revised: Surface normal prediction uses the inverse of cosine similarity between the normalized prediction and ground truth, and is evaluated using mean and median angle distances between the prediction and the ground truth (the lower the better), and the percentage of pixels whose prediction is within the angles of 11 . 25°, 22 . 5° and 30° to the ground truth (Eigen and Fergus, 2015) (the higher the better).

Original: Depth estimation uses the L1 loss, and the absolute and relative errors between the prediction and the ground truth are computed (the lower the better).
Revised: Depth estimation uses the L1 loss, and the absolute and relative errors between the prediction and the ground truth are computed (the lower the better).

Original: In Taskonomy, all the tasks are trained using the same loss as in NYUv2 and directly evaluated by the task-specific loss.
Revised: In Taskonomy, all the tasks are trained using the same loss as in NYUv2 and directly evaluated by the task-specific loss.

Original: Since tasks have multiple evaluation metrics and their value can also be at different scales, we compute a single relative performance 
Revised: Since tasks have multiple evaluation metrics and their value can also be at different scales, we compute a single relative performance metric following (Maninis et al., 2019; Sun et al., 2019). The overall performance is the average of

Original: [Table]
Revised: [Table]

Original: metric following (Maninis et al., 2019; Sun et al., 2019).

Original: The overall performance is the average of the relative performance over all tasks, namely Δ 𝑡 = 1 𝑇 (cid:205) 𝑖 = 1 Δ 𝑡 𝑖 .
Revised: the relative performance over all tasks, namely Δ 𝑡 = 1 𝑇 (cid:205) 𝑖 = 1 Δ 𝑡 𝑖 .

Original: The units for relative performance248 Δ𝑡 𝑖 andΔ 𝑡 are percentage(%).
Revised: The units for relative performance Δ and 𝑡 are percentage

Original: Baselines for Comparison.
Revised: Baselines for Comparison.

Original: Our baselines include both tree-structured MTL methods and general
Revised: Our baselines include both tree-structured MTL methods and general

Original: MTL approaches.
Revised: MTL approaches.

Original: For state-of-the-art tree-structured MTL methods, we compare with What-toShare 3 (Vandenhende et al., 2019), BMTAS 4 (Bruggemann et al., 2020), Learn-to-Branch 5 (Guo et al., 2020), Task-Grouping 5 (Standley et al., 2020).
Revised: For state-of-the-art tree-structured MTL methods, we compare with What-toShare 3 (Vandenhende et al., 2019), BMTAS 4 (Bruggemann et al., 2020), Learn-to-Branch 5 (Guo et al., 2020), Task-Grouping 5 (Standley et al., 2020).

Original: For general MTL approaches, we compare with following baselines: the Single-Task baseline where each task has its own model and is trained independently, popular MTL methods (e.g., Cross-Stitch (Misra et al., 2016), Sluice (Ruder et al., 2019), NDDR-CNN (Gao et al., 2019),
Revised: For general MTL approaches, we compare with following baselines: the Single-Task baseline where each task has its own model and is trained independently, popular MTL methods (e.g., Cross-Stitch (Misra et al., 2016), Sluice (Ruder et al., 2019), NDDR-CNN (Gao et al., 2019),

Original: MTAN (Liu et al., 2019)), and state-of-the-art NAS-based MTL methods (e.g. DEN (Ahn et al., 2019), AdaShare (Sun et al., 2019), AutoMTL (Zhang et al., 2021)).
Revised: MTAN (Liu et al., 2019)), and state-of-the-art NAS-based MTL methods (e.g. DEN (Ahn et al., 2019), AdaShare (Sun et al., 2019), AutoMTL (Zhang et al., 2021)).

Original: We use the same backbone model in all baselines and in our approach for fair comparisons.
Revised: We use the same backbone model in all baselines and in our approach for fair comparisons.

Original: We use Deeplab-ResNet34 (Chen et al., 2017) and MobileNetV2 (Sandler et al., 2018) as the backbone model and the Atrous Spatial Pyramid Pooling (ASPP) architecture as the task-specific head.
Revised: We use Deeplab-ResNet34 (Chen et al., 2017) and MobileNetV2 (Sandler et al., 2018) as the backbone model and the Atrous Spatial Pyramid Pooling (ASPP) architecture as the task-specific head.

Original: Both of them are popular architectures for pixel-wise prediction tasks.
Revised: Both of them are popular architectures for pixel-wise prediction tasks.

Original: The branching points of DeeplabResNet34 are generated by our branching point detector and then further customized to be five according to He et al.
Revised: The branching points of DeeplabResNet34 are generated by our branching point detector and then further customized to be five according to He et al.

Original: (2016) to reduce search space, each computation block corresponding to one
Revised: (2016) to reduce search space, each computation block corresponding to one

Original: ConvBNReLU block or one Residual Block.
Revised: ConvBNReLU block or one Residual Block.

Original: Similarly, MobileNetV2 is split into separate Inverted
Revised: Similarly, MobileNetV2 is split into separate Inverted

Original: Blocks firstly and then its branching points are defined by merging adjacent blocks into five larger ones with similar computation cost measured by FLOPs.
Revised: Blocks firstly and then its branching points are defined by merging adjacent blocks into five larger ones with similar computation cost measured by FLOPs.

Original: 4.2 Performance of Recommended Tree-Structured Multi-Task Models
Revised: 4.2 Performance of Recommended Tree-Structured Multi-Task Models

Original: Table 1 ∼ 2 report the real task performance of the recommended tree-structured multi-task models after training using Deeplab-ResNet34.
Revised: Table 1 ∼ 2 report the real task performance of the recommended tree-structured multi-task models after training using Deeplab-ResNet34.

Original: It reports both absolute values of all evaluation metrics and the relative performance.
Revised: It reports both absolute values of all evaluation metrics and the relative performance.

Original: Results on MobileNetV2 can be found in the Appendix Section ?? .

Original: The first column “Model” lists the index of the recommended models.
Revised: The first column “Model” lists the index of the recommended models.

Original: The specific model structures are shown in the Appendix Section ?? .

Original: Overall, the recommendation of our framework is consistent with the common belief that MTL can achieve higher task accuracy and improved efficiency for each task by leveraging commonalities across related tasks (Caruana, 1997; Ruder, 2017).
Revised: Overall, the recommendation of our framework is consistent with the common belief that MTL can achieve higher task accuracy and improved efficiency for each task by leveraging commonalities across related tasks (Caruana, 1997; Ruder, 2017).

Original: The superiority of our recommender can be observed more clearly in Table 2.
Revised: The superiority of our recommender can be observed more clearly in Table 2.

Original: With different computation budgets (specified by the number of backbone models in column “Com. Budget”), our recommender could always recommend multi-task architectures with high task performance within the computation constraint.
Revised: With different computation budgets (specified by the number of backbone models in column “Com. Budget”), our recommender could always recommend multi-task architectures with high task performance within the computation constraint.

Original: Unlike prior works (Sun et al., 2019; Bruggemann et al., 2020), which have to re-train the whole architecture searching framework when the computational requirement changes, there is no extra effort for our framework to re-predict the top architectures.
Revised: Unlike prior works (Sun et al., 2019; Bruggemann et al., 2020), which have to re-train the whole architecture searching framework when the computational requirement changes, there is no extra effort for our framework to re-predict the top architectures.

Original: [Table] The recommender can suggest top architectures on the fly by filtering out architectures that do not satisfy the given requirement. 
Revised: The recommender can suggest top architectures on the fly by filtering out architectures that do not satisfy the given requirement. [Table]

Original: 4.3 Evaluation of the Task Accuracy Estimator
Revised: 4.3 Evaluation of the Task Accuracy Estimator

Original: [Table]
Revised: [Table]

Original: Our recommender can get a predicted ranking of all the multi-task architectures based on their estimated task performance from the task accuracy estimator.
Revised: Our recommender can get a predicted ranking of all the multi-task architectures based on their estimated task performance from the task accuracy estimator.

Original: To evaluate the predicted ranking, we also get an oracle ranking , by actual training the multi-task architectures.
Revised: To evaluate the predicted ranking, we also get an oracle ranking , by actual training the multi-task architectures.

Original: We use the Pearson correlation coefficient (Pearson’s 𝛾 ) (Benesty et al., 2009) of the predicted ranking and the oracle ranking to evaluate the efficacy of the task accuracy estimator component.
Revised: We use the Pearson correlation coefficient (Pearson’s 𝛾 ) (Benesty et al., 2009) of the predicted ranking and the oracle ranking to evaluate the efficacy of the task accuracy estimator component.

Original: We compare with the correlation of What-toShare (Vandenhende et al., 2019), the only existing branched MTL method which could sort the architectures according to task dissimilarity scores.
Revised: We compare with the correlation of What-toShare (Vandenhende et al., 2019), the only existing branched MTL method which could sort the architectures according to task dissimilarity scores.

Original: Table 3 reports the correlation results.
Revised: Table 3 reports the correlation results.

Original: For reproducibility, the random seed of the experiments is set as 10.
Revised: For reproducibility, the random seed of the experiments is set as 10.

Original: For NYUv2 on MobileNetV2, we also conduct the same experiment with seed 20.
Revised: For NYUv2 on MobileNetV2, we also conduct the same experiment with seed 20.

Original: The range of 𝛾 is [− 1 , 1 ] .
Revised: The range of 𝛾 is [− 1 , 1 ] .

Original: The larger the value of 𝛾 is, the stronger the positive correlation, and the better the predicted ranking.
Revised: The larger the value of 𝛾 is, the stronger the positive correlation, and the better the predicted ranking.

Original: Overall, our estimated architecture ranking has a moderately high correlation (i.e., 0 .
Revised: Overall, our estimated architecture ranking has a moderately high correlation (i.e., 0 .

Original: 4 ≤ 𝛾 < 0 .
Revised: 4 ≤ 𝛾 < 0 .

Original: 7) or even very strong correlation (i.e., 0 .
Revised: 7) or even very strong correlation (i.e., 0 .

Original: 7 ≤ 𝛾 < 0 . 9) with the oracle ranking according to the interpretation of Pearson’s300 𝛾 (Akoglu, 2018), which demonstrates the reliability of the task accuracy estimator and the effectiveness of our recommender.
Revised: 7 ≤ 𝛾 < 0 . 9) with the oracle ranking according to the interpretation of Pearson’s 𝛾 (Akoglu, 2018), which demonstrates the reliability of the task accuracy estimator and the effectiveness of our recommender.

Original: In contrast, What-to-Share produces negative correlations, indicating that their estimations from task dissimilarity are unreliable.
Revised: In contrast, What-to-Share produces negative correlations, indicating that their estimations from task dissimilarity are unreliable.

Original: Compared with What-to-Share, our recommender improves the correlation significantly.
Revised: Compared with What-to-Share, our recommender improves the correlation significantly.

Original: 4.4 Comparison with State-of-the-Art MTL Methods
Revised: 4.4 Comparison with State-of-the-Art MTL Methods

Original: Table 4 summarizes the comparisons with state-of-the-art MTL methods for Taskonomy on DeeplabResNet34.
Revised: Table 4 summarizes the comparisons with state-of-the-art MTL methods for Taskonomy on DeeplabResNet34.

Original: Results on other datasets and backbone models are included in the Appendix Section ?? .
Revised: Table 5 and 6 report the results on MobileNetV2.

Original: Generally, the best multi-task architectures suggested by our recommender could achieve competitive or even higher overall task performance as indicated by the Δ 𝑡 columns.
Revised: Generally, the best multi-task architectures suggested by our recommender could achieve competitive or even higher overall task performance as indicated by the Δ 𝑡 columns.

Original: Our work is closest to What-to-Share which also ranks all the candidate multi-task architectures and outperforms it by 4.8% in terms of the overall task performance.
Revised: Our work is closest to What-to-Share which also ranks all the candidate multi-task architectures and outperforms it by 4.8% in terms of the overall task performance.

Original: Task-Grouping focuses on deciding how to split the tasks into groups according to the given computation budget so that one group will share the entire backbone model.
Revised: Task-Grouping focuses on deciding how to split the tasks into groups according to the given computation budget so that one group will share the entire backbone model.

Original: Compared to Task-Grouping , our recommender yields better branching models under the same budget.
Revised: Compared to Task-Grouping , our recommender yields better branching models under the same budget.

Original: For instance, when the budget is three models, our top-1 multi-task architecture could achieve higher task performance (3.3% vs 2.4%) with lower computation cost (-41.93% vs -40%) and number of parameters (-41.27% vs -40%) than
Revised: For instance, when the budget is three models, our top-1 multi-task architecture could achieve higher task performance (3.3% vs 2.4%) with lower computation cost (-41.93% vs -40%) and number of parameters (-41.27% vs -40%) than

Original: Task-Grouping.
Revised: Task-Grouping.

Revised: Compared with manually-design multi-task architectures, Cross-Stitch , Sluice , NDDR-CNN , and MTAN , which usually consist of separate networks for each task and define a mechanism for

Original: [Table]
Revised: [Table]

Original: Compared with manually-design multi-task architectures, Cross-Stitch , Sluice , NDDR-CNN , and MTAN , which usually consist of separate networks for each task and define a mechanism for feature sharing between independent networks, our recommended architectures perform higher task performance (4.9% vs -1.6%/-2.8%/-1.5%/-3.0%) with computation cost (-11.31% vs 0%/8.38%/-10.55%) and the number of parameters reduction (-7.90% vs 0%/8.20%/-9.80%).
Revised: [Table] feature sharing between independent networks, our recommended architectures perform higher task performance (4.9% vs -1.6%/-2.8%/-1.5%/-3.0%) with computation cost (-11.31% vs 0%/8.38%/- 10.55%) and the number of parameters reduction (-7.90% vs 0%/8.20%/-9.80%).

Original: We also compare with NAS-based methods, including NAS-based branched MTL methods such as Learn-to-Branch and BMTAS , and NAS-based general MTL approaches such as DEN , AdaShare , and AutoMTL .
Revised: We also compare with NAS-based methods, including NAS-based branched MTL methods such as Learn-to-Branch and BMTAS , and NAS-based general MTL approaches such as DEN , AdaShare , and AutoMTL .

Original: Learn-to-Branch and BMTAS explore the same tree-structured architecture design space as our recommender.
Revised: Learn-to-Branch and BMTAS explore the same tree-structured architecture design space as our recommender.

Original: However, since they resort to integrating space searching with network training, the searched multi-task models are usually sub-optimal.
Revised: However, since they resort to integrating space searching with network training, the searched multi-task models are usually sub-optimal.

Original: Instead, our recommender could overcome the limitation to identify multi-task architectures with higher task performance, 5.4% higher than Learn-to-Branch, and 0.3%/4.1% higher than BMTAS on NYUv2 and Taskonomy using
Revised: Instead, our recommender could overcome the limitation to identify multi-task architectures with higher task performance, 5.4% higher than Learn-to-Branch, and 0.3%/4.1% higher than BMTAS on NYUv2 and Taskonomy using

Original: MobileNetV2 as shown in Table ?? and ??
Revised: MobileNetV2 as shown in Table 5 and 6.

Original: in the Appendix.

Original: When comparing to DEN, AdaShare, and AutoMTL, our recommender identifies multi-task architectures with competitive task performance (4.9% vs -10.4%/0.8%/4.7%), even though the search space of those methods are larger and more complex than our tree-structured multi-task model space.
Revised: When comparing to DEN, AdaShare, and AutoMTL, our recommender identifies multi-task architectures with competitive task performance (4.9% vs -10.4%/0.8%/4.7%), even though the search space of those methods are larger and more complex than our tree-structured multi-task model space [Table] [Table]

Original: 5 Conclusion
Revised: 5 Conclusion

Original: This paper proposes a tree-structured multi-task model recommender that predicts the top- 𝑘 architectures with high task performance given a set of tasks, an arbitrary CNN backbone model, and a user-specified computation budget.
Revised: This paper proposes a tree-structured multi-task model recommender that predicts the top- 𝑘 architectures with high task performance given a set of tasks, an arbitrary CNN backbone model, and a user-specified computation budget.

Original: Our recommender consists of three key components, a branching point detector that automatically detects branching points in any given CNN backbone model, a design space enumerator that enumerates all the multi-task architecture in the design space, and a task accuracy estimator that predicts the task performance of multi-task architectures without performing actual training.
Revised: Our recommender consists of three key components, a branching point detector that automatically detects branching points in any given CNN backbone model, a design space enumerator that enumerates all the multi-task architecture in the design space, and a task accuracy estimator that predicts the task performance of multi-task architectures without performing actual training.

Original: Experiments on popular MTL benchmarks demonstrate the superiority and reliability of our recommender compared with state-of-the-art approaches.
Revised: Experiments on popular MTL benchmarks demonstrate the superiority and reliability of our recommender compared with state-of-the-art approaches.

Original: Limitations and Broader Impact Statement.
Revised: Limitations and Broader Impact Statement.

Original: Our research facilitates the adoption of deep learning techniques to solve many tasks at once in resource-constraint scenarios.
Revised: Our research facilitates the adoption of deep learning techniques to solve many tasks at once in resource-constraint scenarios.

Original: It also promotes the leverage of multi-task learning to increase task performance and computation efficiency.
Revised: It also promotes the leverage of multi-task learning to increase task performance and computation efficiency.

Original: It has a positive impact on applications that tackle multiple tasks such as environment perceptions for autonomous vehicles and human-computer interactions in robotic, mobile, and IoT applications.
Revised: It has a positive impact on applications that tackle multiple tasks such as environment perceptions for autonomous vehicles and human-computer interactions in robotic, mobile, and IoT applications.

Original: The negative social impact of our research is difficult to predict since it shares the same pitfalls with general deep learning techniques that suffer from dataset bias, adversarial attacks, fairness, etc.
Revised: The negative social impact of our research is difficult to predict since it shares the same pitfalls with general deep learning techniques that suffer from dataset bias, adversarial attacks, fairness, etc.