Original: NASOA: T OWARDS F ASTER T ASK - ORIENTED O NLINE F INE - TUNING
Revised: NASOA: T OWARDS F ASTER T ASK - ORIENTED O NLINE F INE - TUNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A
Revised: A

Original: BSTRACT
Revised: BSTRACT

Original: Fine-tuning from pre-trained ImageNet models has been a simple, effective, and popular approach for various computer vision tasks.
Revised: Fine-tuning from pre-trained ImageNet models has been a simple, effective, and popular approach for various computer vision tasks.

Original: The common practice of ﬁnetuning is to adopt a default hyperparameter setting with a ﬁxed pre-trained model, while both of them are not optimized for speciﬁc tasks and time constraints.
Revised: The common practice of ﬁnetuning is to adopt a default hyperparameter setting with a ﬁxed pre-trained model, while both of them are not optimized for speciﬁc tasks and time constraints.

Original: Moreover, in cloud computing or GPU clusters where the tasks arrive sequentially in a stream, faster online ﬁne-tuning is a more desired and realistic strategy for saving money, energy consumption, and CO2 emission.
Revised: Moreover, in cloud computing or GPU clusters where the tasks arrive sequentially in a stream, faster online ﬁne-tuning is a more desired and realistic strategy for saving money, energy consumption, and CO2 emission.

Original: In this paper, we propose a joint Neural Architecture Search and Online Adaption framework named NASOA towards a faster task-oriented ﬁne-tuning upon the request of users.
Revised: In this paper, we propose a joint Neural Architecture Search and Online Adaption framework named NASOA towards a faster task-oriented ﬁne-tuning upon the request of users.

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: NASOA ﬁrst adopts an ofﬂine NAS to identify a group of training-efﬁcient networks to form a pretrained model zoo.
Revised: NASOA ﬁrst adopts an ofﬂine NAS to identify a group of training-efﬁcient networks to form a pretrained model zoo.

Original: We propose a novel joint block and macro level search space to enable a ﬂexible and efﬁcient search.
Revised: We propose a novel joint block and macro level search space to enable a ﬂexible and efﬁcient search.

Original: Then, by estimating ﬁne-tuning performance via an adaptive model by accumulating experience from the past tasks, an online schedule generator is proposed to pick up the most suitable model and generate a personalized training regime with respect to each desired task in a one-shot fashion.
Revised: Then, by estimating ﬁne-tuning performance via an adaptive model by accumulating experience from the past tasks, an online schedule generator is proposed to pick up the most suitable model and generate a personalized training regime with respect to each desired task in a one-shot fashion.

Original: The resulting model zoo 1 is more training efﬁcient than SOTA NAS models, e.g. 6x faster than RegNetY-16GF, and 1.7x faster than EfﬁcientNetB3.
Revised: The resulting model zoo 1 is more training efﬁcient than SOTA NAS models, e.g. 6x faster than RegNetY-16GF, and 1.7x faster than EfﬁcientNetB3.

Original: Experiments on multiple datasets also show that NASOA achieves much better ﬁne-tuning results, i.e. improving around 2.1% accuracy than the best performance in RegNet series under various time constraints and tasks; 40x faster compared to the BOHB method.
Revised: Experiments on multiple datasets also show that NASOA achieves much better ﬁne-tuning results, i.e. improving around 2.1% accuracy than the best performance in RegNet series under various time constraints and tasks; 40x faster compared to the BOHB method.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Fine-tuning using pre-trained models becomes the de-facto standard in the ﬁeld of computer vision because of its impressive results on various downstream tasks such as ﬁne-grained image classiﬁcation (Nilsback & Zisserman, 2008; Welinder et al., 2010), object detection (He et al., 2019; Jiang et al., 2018;
Revised: Fine-tuning using pre-trained models becomes the de-facto standard in the ﬁeld of computer vision because of its impressive results on various downstream tasks such as ﬁne-grained image classiﬁcation (Nilsback & Zisserman, 2008; Welinder et al., 2010), object detection (He et al., 2019; Jiang et al., 2018;

Original: Xu et al., 2019) and segmentation (Chen et al., 2017; Liu et al., 2019).
Revised: Xu et al., 2019) and segmentation (Chen et al., 2017; Liu et al., 2019).

Original: Kornblith et al.
Revised: Kornblith et al.

Original: ; He et al.
Revised: ; He et al.

Original: (2019) veriﬁed that ﬁne-tuning pre-trained networks outperform training from scratch.
Revised: (2019) veriﬁed that ﬁne-tuning pre-trained networks outperform training from scratch.

Original: It can further help to avoid over-ﬁtting (Cui et al., 2018) as well as reduce training time signiﬁcantly (He et al., 2019).
Revised: It can further help to avoid over-ﬁtting (Cui et al., 2018) as well as reduce training time signiﬁcantly (He et al., 2019).

Original: Due to those merits, many cloud computing and AutoML pipelines provide ﬁne-tuning services for an online stream of upcoming users with new data, different tasks and time limits.
Revised: Due to those merits, many cloud computing and AutoML pipelines provide ﬁne-tuning services for an online stream of upcoming users with new data, different tasks and time limits.

Original: In order to save the user’s time, money, energy consumption, or even CO2 emission, an efﬁcient online automated ﬁne-tuning framework is practically useful and in great demand.
Revised: In order to save the user’s time, money, energy consumption, or even CO2 emission, an efﬁcient online automated ﬁne-tuning framework is practically useful and in great demand.

Original: Thus, in this work, we propose to explore the problem of faster online ﬁne-tuning.
Revised: Thus, in this work, we propose to explore the problem of faster online ﬁne-tuning.

Original: The conventional practice of ﬁne-tuning is to adopt a set of predeﬁned hyperparameters for training a predeﬁned model (Li et al., 2020).
Revised: The conventional practice of ﬁne-tuning is to adopt a set of predeﬁned hyperparameters for training a predeﬁned model (Li et al., 2020).

Original: It has three drawbacks in the current online setting: 1) The design of the backbone model is not optimized for the upcoming ﬁne-tuning task and the selection of the backbone model is not data-speciﬁc.
Revised: It has three drawbacks in the current online setting: 1) The design of the backbone model is not optimized for the upcoming ﬁne-tuning task and the selection of the backbone model is not data-speciﬁc.

Original: A default setting of hyperparameters may not be optimal across tasks and the training settings may not meet the time constraints provided by users.
Revised: A default setting of hyperparameters may not be optimal across tasks and the training settings may not meet the time constraints provided by users.

Original: 3) With the incoming tasks, the regular diagram is not suitable for this online setting since it cannot memorize and accumulate experience from the past ﬁne-tuning tasks.
Revised: 3) With the incoming tasks, the regular diagram is not suitable for this online setting since it cannot memorize and accumulate experience from the past ﬁne-tuning tasks.

Original: Thus, we propose to decouple our faster ﬁne-tuning problem into two parts: ﬁnding efﬁcient ﬁne-tuning networks and generating optimal ﬁne-tuning schedules pertinent to speciﬁc time constraints in an online learning fashion.
Revised: Thus, we propose to decouple our faster ﬁne-tuning problem into two parts: ﬁnding efﬁcient ﬁne-tuning networks and generating optimal ﬁne-tuning schedules pertinent to speciﬁc time constraints in an online learning fashion.

Original: Recently, Neural Architecture Search (NAS) algorithms demonstrate promising results on discovering top-accuracy architectures, which surpass the performance of hand-crafted networks and saves human’s efforts (Zoph et al., 2018; Liu et al., 2018a;b; Radosavovic et al., 2019; Tan et al., 2019b;
Revised: Recently, Neural Architecture Search (NAS) algorithms demonstrate promising results on discovering top-accuracy architectures, which surpass the performance of hand-crafted networks and saves human’s efforts (Zoph et al., 2018; Liu et al., 2018a;b; Radosavovic et al., 2019; Tan et al., 2019b;

Original: Real et al., 2019a; Tan & Le, 2019; Yao et al., 2020).
Revised: Real et al., 2019a; Tan & Le, 2019; Yao et al., 2020).

Original: However, those NAS works usually focus on inference time/FLOPS optimization and their search space is not ﬂexible enough which cannot guarantee the optimality for fast ﬁne-tuning.
Revised: However, those NAS works usually focus on inference time/FLOPS optimization and their search space is not ﬂexible enough which cannot guarantee the optimality for fast ﬁne-tuning.

Original: In contrast, we resort to developing a NAS scheme with a novel ﬂexible search space for fast ﬁne-tuning.
Revised: In contrast, we resort to developing a NAS scheme with a novel ﬂexible search space for fast ﬁne-tuning.

Original: On the other hand, hyperparameter optimization (HPO) methods such as grid search (Bergstra & Bengio, 2012), Bayesian optimization (BO) (Strubell et al., 2019a; Mendoza et al., 2016), and BOHB (Falkner et al., 2018) are used in deep learning and achieve good performance.
Revised: On the other hand, hyperparameter optimization (HPO) methods such as grid search (Bergstra & Bengio, 2012), Bayesian optimization (BO) (Strubell et al., 2019a; Mendoza et al., 2016), and BOHB (Falkner et al., 2018) are used in deep learning and achieve good performance.

Original: However, those search-based methods are computationally expensive and require iterative “trial and error”, which violate our goal for faster adaptation time.
Revised: However, those search-based methods are computationally expensive and require iterative “trial and error”, which violate our goal for faster adaptation time.

Original: In this work, we propose a novel Neural Architecture Search and Online Adaption framework named
Revised: In this work, we propose a novel Neural Architecture Search and Online Adaption framework named

Original: NASOA.
Revised: NASOA.

Original: First, we conduct an ofﬂine NAS for generating an efﬁcient ﬁne-tuning model zoo.
Revised: First, we conduct an ofﬂine NAS for generating an efﬁcient ﬁne-tuning model zoo.

Original: We design a novel block-level and macro-structure search space to allow a ﬂexible choice of the networks.
Revised: We design a novel block-level and macro-structure search space to allow a ﬂexible choice of the networks.

Original: Once the efﬁcient training model zoo is created ofﬂine NAS by Pareto optimal models, the online user can enjoy the beneﬁt of those efﬁcient training networks without any marginal cost.
Revised: Once the efﬁcient training model zoo is created ofﬂine NAS by Pareto optimal models, the online user can enjoy the beneﬁt of those efﬁcient training networks without any marginal cost.

Original: We then propose an online learning algorithm with an adaptive predictor to modeling the relation between different hyperparameter, model, dataset meta-info and the ﬁnal ﬁne-tuning performance.
Revised: We then propose an online learning algorithm with an adaptive predictor to modeling the relation between different hyperparameter, model, dataset meta-info and the ﬁnal ﬁne-tuning performance.

Original: The ﬁnal training schedule is generated directly from selecting the ﬁne-tuning regime with the best predicted performance.
Revised: The ﬁnal training schedule is generated directly from selecting the ﬁne-tuning regime with the best predicted performance.

Original: Beneﬁting from the experience accumulation via online learning, the diversity of the data and the increasing results can further continuously improve our regime generator.
Revised: Beneﬁting from the experience accumulation via online learning, the diversity of the data and the increasing results can further continuously improve our regime generator.

Original: Our method behaves in a one-shot fashion and doesn’t involve additional searching cost as HPO, endowing the capability of providing various training regimes under different time constraints.
Revised: Our method behaves in a one-shot fashion and doesn’t involve additional searching cost as HPO, endowing the capability of providing various training regimes under different time constraints.

Original: Extensive experiments are conducted on multiple widely used ﬁne-tuning datasets.
Revised: Extensive experiments are conducted on multiple widely used ﬁne-tuning datasets.

Original: The searched model zoo ET-NAS is more training efﬁcient than SOTA ImageNet models, e.g. 5x training faster than RegNetY-16GF, and 1.7x faster than EfﬁcientNetB3.
Revised: The searched model zoo ET-NAS is more training efﬁcient than SOTA ImageNet models, e.g. 5x training faster than RegNetY-16GF, and 1.7x faster than EfﬁcientNetB3.

Original: Moreover, by using the whole NASOA, our online algorithm achieves superior ﬁne-tuning results in terms of both accuracy and ﬁne-tuning speed, i.e. improving around 2.1% accuracy than the best performance in RegNet series under various tasks; saving 40x computational cost comparing to the BOHB method.
Revised: Moreover, by using the whole NASOA, our online algorithm achieves superior ﬁne-tuning results in terms of both accuracy and ﬁne-tuning speed, i.e. improving around 2.1% accuracy than the best performance in RegNet series under various tasks; saving 40x computational cost comparing to the BOHB method.

Original: In summary, our contributions are summarized as follows:
Revised: In summary, our contributions are summarized as follows:

Original: • To the best of our knowledge, we make the ﬁrst effort to propose a faster ﬁne-tuning pipeline that seamlessly combines the training-efﬁcient NAS and online adaption algorithm.
Revised: • To the best of our knowledge, we make the ﬁrst effort to propose a faster ﬁne-tuning pipeline that seamlessly combines the training-efﬁcient NAS and online adaption algorithm.

Original: Our NASOA can effectively generate a personalized ﬁne-tuning schedule of each desired task via an adaptive model for accumulating experience from the past tasks.
Revised: Our NASOA can effectively generate a personalized ﬁne-tuning schedule of each desired task via an adaptive model for accumulating experience from the past tasks.

Original: • The proposed novel joint block/macro level search space enables a ﬂexible and efﬁcient search.
Revised: • The proposed novel joint block/macro level search space enables a ﬂexible and efﬁcient search.

Original: The resulting model zoo ET-NAS is more training efﬁcient than very strong ImageNet SOTA models e.g. EfﬁcientNet, RegNet.
Revised: The resulting model zoo ET-NAS is more training efﬁcient than very strong ImageNet SOTA models e.g. EfﬁcientNet, RegNet.

Original: All the ET-NAS models have been released to help the community skipping the computation-heavy NAS stage and directly enjoy the beneﬁt of NASOA.
Revised: All the ET-NAS models have been released to help the community skipping the computation-heavy NAS stage and directly enjoy the beneﬁt of NASOA.

Original: • The whole NASOA pipeline achieves much better ﬁne-tuning results in terms of both accuracy and ﬁne-tuning efﬁciency than current ﬁne-tuning best practice and HPO method,e.g. BOHB.
Revised: • The whole NASOA pipeline achieves much better ﬁne-tuning results in terms of both accuracy and ﬁne-tuning efﬁciency than current ﬁne-tuning best practice and HPO method,e.g. BOHB.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Neural Architecture Search (NAS).
Revised: Neural Architecture Search (NAS).

Original: The goal of NAS is to automatically optimize network architecture and release human effort from this handcraft network architecture engineering.
Revised: The goal of NAS is to automatically optimize network architecture and release human effort from this handcraft network architecture engineering.

Original: Most previous works (Liu et al., 2018b; Cai et al., 2019b;
Revised: Most previous works (Liu et al., 2018b; Cai et al., 2019b;

Original: Liu et al., 2018a; Tan et al., 2019a; Xie et al., 2019; Howard et al., 2019) aim at searching for CNN architectures with better inference and fewer
Revised: Liu et al., 2018a; Tan et al., 2019a; Xie et al., 2019; Howard et al., 2019) aim at searching for CNN architectures with better inference and fewer

Original: FLOPS.
Revised: FLOPS.

Original: Baker et al.
Revised: Baker et al.

Original: ; Cai et al.
Revised: ; Cai et al.

Original: ; Zhong et al.
Revised: ; Zhong et al.

Original: (2018) apply reinforcement learning to train an RNN controller to generate a cell architecture.
Revised: (2018) apply reinforcement learning to train an RNN controller to generate a cell architecture.

Original: Liu et al.
Revised: Liu et al.

Original: (2018b);
Revised: (2018b);

Original: Xie et al.
Revised: Xie et al.

Original: ; Cai et al.
Revised: ; Cai et al.

Original: (2019b) try to search a cell structure by weight-sharing and differentiable optimization.
Revised: (2019b) try to search a cell structure by weight-sharing and differentiable optimization.

Original: Tan & Le (2019) use a grid search for an efﬁcient network by altering the depth/width of the network with a ﬁxed block structure.
Revised: Tan & Le (2019) use a grid search for an efﬁcient network by altering the depth/width of the network with a ﬁxed block structure.

Original: On the contrary, our NAS focuses creating an efﬁcient training model zoo for fast ﬁne-tuning.
Revised: On the contrary, our NAS focuses creating an efﬁcient training model zoo for fast ﬁne-tuning.

Original: Moreover, the existing search space design cannot meet the purpose of our search.
Revised: Moreover, the existing search space design cannot meet the purpose of our search.

Original: Generating Hyperparameters for Fine-tuning.
Revised: Generating Hyperparameters for Fine-tuning.

Original: HPO methods such as Bayesian optimization (BO) (Strubell et al., 2019a;
Revised: HPO methods such as Bayesian optimization (BO) (Strubell et al., 2019a;

Original: Mendoza et al., 2016), BOHB (Falkner et al., 2018) achieves very promising result but require a lot of computational resources which is contradictory to our original
Revised: Mendoza et al., 2016), BOHB (Falkner et al., 2018) achieves very promising result but require a lot of computational resources which is contradictory to our original

Original: [Figure]
Revised: [Figure]

Original: Offline Neural Architecture Search
Revised: Offline Neural Architecture Search

Original: [Figure]
Revised: [Figure]

Original: Online Cross-task Adaption
Revised: Online Cross-task Adaption

Original: [Figure]
Revised: [Figure]

Original: objective of efﬁcient ﬁne-tuning.
Revised: objective of efﬁcient ﬁne-tuning.

Original: On the other hand, limited works discuss the model selection and
Revised: On the other hand, limited works discuss the model selection and

Original: HPO for ﬁne-tuning.
Revised: HPO for ﬁne-tuning.

Original: Kornblith et al.
Revised: Kornblith et al.

Original: (2019) ﬁnds that ImageNet accuracy and ﬁne-tuning accuracy of different models are highly correlated.
Revised: (2019) ﬁnds that ImageNet accuracy and ﬁne-tuning accuracy of different models are highly correlated.

Original: Li et al.
Revised: Li et al.

Original: ; Achille et al.
Revised: ; Achille et al.

Original: (2019) suggest that the optimal hyperparameters and model for ﬁne-tuning should be both dataset dependent and domain similarity dependent (Cui et al., 2018).
Revised: (2019) suggest that the optimal hyperparameters and model for ﬁne-tuning should be both dataset dependent and domain similarity dependent (Cui et al., 2018).

Original: HyperStar
Revised: HyperStar

Original: (Mittal et al., 2020) is a concurrent HPO work demonstrating that a performance predictor can effectively generate good hyper-parameters for a single model.
Revised: (Mittal et al., 2020) is a concurrent HPO work demonstrating that a performance predictor can effectively generate good hyper-parameters for a single model.

Original: However, those works don’t give an explicit solution about how to perform ﬁnetuning in a more practical online scenario.
Revised: However, those works don’t give an explicit solution about how to perform ﬁnetuning in a more practical online scenario.

Original: In this work, we take the advantage of online learning (Hoi et al., 2018;
Revised: In this work, we take the advantage of online learning (Hoi et al., 2018;

Original: Sahoo et al., 2017) to build a schedule generator, which allows us to memorize the past training history and provide up-and-coming training regimes for new coming tasks on the ﬂy.
Revised: Sahoo et al., 2017) to build a schedule generator, which allows us to memorize the past training history and provide up-and-coming training regimes for new coming tasks on the ﬂy.

Original: Besides, we introduce the NAS model zoo to further push up the speed and performance.
Revised: Besides, we introduce the NAS model zoo to further push up the speed and performance.

Original:  T HE P ROPOSED A PPROACH
Revised:  T HE P ROPOSED A PPROACH

Original: The goal of this paper is to develop an online ﬁne-tuning pipeline to facilitate a fast continuous cross-task model adaption.
Revised: The goal of this paper is to develop an online ﬁne-tuning pipeline to facilitate a fast continuous cross-task model adaption.

Original: By the preliminary experiments in Section 4.1, we conﬁrm that the model architectures and hyperparameters such as the learning rate and frozen stages will greatly inﬂuence the accuracy and speed of the ﬁne-tuning program.
Revised: By the preliminary experiments in Section 4.1, we conﬁrm that the model architectures and hyperparameters such as the learning rate and frozen stages will greatly inﬂuence the accuracy and speed of the ﬁne-tuning program.

Original: Thus, our NASOA includes two parts as shown in the Figure 1: 1) Searching a group of neural architectures with good accuracy and fast training speed to create a pretrained model zoo; 2) Designing an online task-oriented algorithm to generate an efﬁcient ﬁne-tuning regime with the most suitable model under user’s time constraint.
Revised: Thus, our NASOA includes two parts as shown in the Figure 1: 1) Searching a group of neural architectures with good accuracy and fast training speed to create a pretrained model zoo; 2) Designing an online task-oriented algorithm to generate an efﬁcient ﬁne-tuning regime with the most suitable model under user’s time constraint.

Original: 3. C REATING AN E FFICIENT T RAINING M ODEL Z OO (ET-NAS) BY NAS
Revised: 3. C REATING AN E FFICIENT T RAINING M ODEL Z OO (ET-NAS) BY NAS

Original: The commonly used hand-craft backbones for ﬁne-tuning including MobileNet (Sandler et al., 2018), ResNet (He et al., 2016), and ResNeXt (Xie et al., 2017).
Revised: The commonly used hand-craft backbones for ﬁne-tuning including MobileNet (Sandler et al., 2018), ResNet (He et al., 2016), and ResNeXt (Xie et al., 2017).

Original: Recently, some state-of-the-art backbone series such as RegNet (Radosavovic et al., 2020), and EfﬁcientNet (Tan et al., 2019b) are developed by automated algorithms for higher accuracy and faster inference speed.
Revised: Recently, some state-of-the-art backbone series such as RegNet (Radosavovic et al., 2020), and EfﬁcientNet (Tan et al., 2019b) are developed by automated algorithms for higher accuracy and faster inference speed.

Original: However, the objective of our NAS is to ﬁnd a group of models with good model generalization ability and training speed.
Revised: However, the objective of our NAS is to ﬁnd a group of models with good model generalization ability and training speed.

Original: Suggested by Kornblith et al.
Revised: Suggested by Kornblith et al.

Original: (2019), the model ﬁne-tuning accuracy (model generalization ability) has a strong correlation between ImageNet accuracy ( r = 0 . 96 ).
Revised: (2019), the model ﬁne-tuning accuracy (model generalization ability) has a strong correlation between ImageNet accuracy ( r = 0 . 96 ).

Original: Meanwhile, the training speed can be measured by the step time of each training iteration.
Revised: Meanwhile, the training speed can be measured by the step time of each training iteration.

Original: Thus, our NAS can be formulated by a multi-objective optimization problem (MOOP) on the search space S given by:
Revised: Thus, our NAS can be formulated by a multi-objective optimization problem (MOOP) on the search space S given by:

Original: [Equation]
Revised: [Equation]

Original: (1) where A is the architecture, acc( . ) is the Top-1 accuracy on ImageNet, T s ( . ) is the average step time of one iteration, and T m is the maximum step time allowed.
Revised: (1) where A is the architecture, acc( . ) is the Top-1 accuracy on ImageNet, T s ( . ) is the average step time of one iteration, and T m is the maximum step time allowed.

Original: The step time is deﬁned to be the total time of one iteration, including forward/backward propagation, and parameter update.
Revised: The step time is deﬁned to be the total time of one iteration, including forward/backward propagation, and parameter update.

Original: Search Space Design is extremely important (Radosavovic et al., 2020).
Revised: Search Space Design is extremely important (Radosavovic et al., 2020).

Original: As shown in Figure 2, to enable an efﬁcient model, we propose a novel ﬂexible joint block-level and macro-level search space to enable simple to complex block design and ﬁne adjustment of the computation allocation on each stage.
Revised: As shown in Figure 2, to enable an efﬁcient model, we propose a novel ﬂexible joint block-level and macro-level search space to enable simple to complex block design and ﬁne adjustment of the computation allocation on each stage.

Original: Unlike existing topological cell-level search space such as DARTS (Liu et al., 2018b),
Revised: Unlike existing topological cell-level search space such as DARTS (Liu et al., 2018b),

Original: AmoebaNet(Real et al., 2019a), and NASBench101(Dong & Yang, 2019), ours is more compact and avoids redundant skip-connections which have great memory access cost (MAC).
Revised: AmoebaNet(Real et al., 2019a), and NASBench101(Dong & Yang, 2019), ours is more compact and avoids redundant skip-connections which have great memory access cost (MAC).

Original: Our block-level search space is more ﬂexible to adjust the width, depth (for each stage), when to down-sample/raise the channels.
Revised: Our block-level search space is more ﬂexible to adjust the width, depth (for each stage), when to down-sample/raise the channels.

Original: In contrast, EfﬁcientNet only scales up/down the total width and depth by a ﬁxed allocation ratio, and RegNet cannot change the number/type of operations in each block.
Revised: In contrast, EfﬁcientNet only scales up/down the total width and depth by a ﬁxed allocation ratio, and RegNet cannot change the number/type of operations in each block.

Original: Macro-
Revised: Macro-

Original: [Figure]
Revised: [Figure]

Original: Search Space
Revised: Search Space

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Stage-2: 1-10 Blocks
Revised: Stage-2: 1-10 Blocks

Original: Stage-3: 1-10 Blocks
Revised: Stage-3: 1-10 Blocks

Original: Stage-4: 1-10 Blocks
Revised: Stage-4: 1-10 Blocks

Original: Channel Size: C
Revised: Channel Size: C

Original: C C
Revised: C C

Original: C
Revised: C

Original: C C 2C 2C 2C 2C 2C 4C 4C
Revised: C C 2C 2C 2C 2C 2C 4C 4C

Original: Block-level Search Space
Revised: Block-level Search Space

Original: Search For: Number, types and channels of Operations, Number, place, and types of Additional Skip Connections
Revised: Search For: Number, types and channels of Operations, Number, place, and types of Additional Skip Connections

Original: I npu t
Revised: I npu t

Original: O P O P
Revised: O P O P

Original: O P
Revised: O P

Original: O u t pu t
Revised: O u t pu t

Original: Residual
Revised: Residual

Original: [Figure]
Revised: [Figure]

Original: C C
Revised: C C

Original: C
Revised: C

Original: Block-level Search Space.
Revised: Block-level Search Space.

Original: We consider a search space based on 1-3 successive nodes of 5 different operations.
Revised: We consider a search space based on 1-3 successive nodes of 5 different operations.

Original: Three skip connections with one ﬁxed residual connection are searched.
Revised: Three skip connections with one ﬁxed residual connection are searched.

Original: Element-wise add or channel-wise concat is chosen to combine the features for the skip-connections.
Revised: Element-wise add or channel-wise concat is chosen to combine the features for the skip-connections.

Original: For each selected operation, we also search for the ratio of changing channel size
Revised: For each selected operation, we also search for the ratio of changing channel size

Original: Note that it can cover many popular block designs such as Bottleneck (He et al., 2016), ResNeXt (Xie et al., 2017) and MB block (Sandler et al., 2018).
Revised: Note that it can cover many popular block designs such as Bottleneck (He et al., 2016), ResNeXt (Xie et al., 2017) and MB block (Sandler et al., 2018).

Original: It consists of 5 . 4 × 10 6 unique blocks.
Revised: It consists of 5 . 4 × 10 6 unique blocks.

Original: Macro-level Search Space.
Revised: Macro-level Search Space.

Original: Allocation computation over different stages is crucial for a backbone (Liang et al., 2020).
Revised: Allocation computation over different stages is crucial for a backbone (Liang et al., 2020).

Original: Early-stage feature maps in one backbone are larger which captures texture details, while late-stage feature maps are smaller which are more discriminative (Li et al., 2018).
Revised: Early-stage feature maps in one backbone are larger which captures texture details, while late-stage feature maps are smaller which are more discriminative (Li et al., 2018).

Original: Therefore, for macro-level search space, we design a ﬂexible search space to ﬁnd the optimal channel size (width), depth (total number of blocks), when to down-sample, and when to raise the channels.
Revised: Therefore, for macro-level search space, we design a ﬂexible search space to ﬁnd the optimal channel size (width), depth (total number of blocks), when to down-sample, and when to raise the channels.

Original: Our macro-level structure consists of 4 ﬂexible stages.
Revised: Our macro-level structure consists of 4 ﬂexible stages.

Original: The spatial size of the stages is gradually down-sampled with factor 2.
Revised: The spatial size of the stages is gradually down-sampled with factor 2.

Original: In each stage, we stack a number of block architectures.
Revised: In each stage, we stack a number of block architectures.

Original: The positions of the doubling channel block are also ﬂexible.
Revised: The positions of the doubling channel block are also ﬂexible.

Original: This search space consists of 1 . 5 × 10 7 unique architectures.
Revised: This search space consists of 1 . 5 × 10 7 unique architectures.

Original: Details of the search space and its encodings can be found in Appendix B.1.
Revised: Details of the search space and its encodings can be found in Appendix B.1.

Original: Multi-objective Searching Algorithm .
Revised: Multi-objective Searching Algorithm .

Original: For MOOP in Eq 1, we deﬁne architecture A 1 dominates
Revised: For MOOP in Eq 1, we deﬁne architecture A 1 dominates

Original: A 2 if (i) A 1 is no worse than A 2 in all objectives; (ii) A 1 is strictly better than A 2 in at least one objective.
Revised: A 2 if (i) A 1 is no worse than A 2 in all objectives; (ii) A 1 is strictly better than A 2 in at least one objective.

Original: A ∗ is Pareto optimal if there is no other A that dominate A ∗ .
Revised: A ∗ is Pareto optimal if there is no other A that dominate A ∗ .

Original: The set of all Pareto optimal architectures constitutes the Pareto front .
Revised: The set of all Pareto optimal architectures constitutes the Pareto front .

Original: To solve this MOOP problem, we modify a well-known method named Elitist Non-Dominated sorting genetic algorithm (NSGA-II) (Deb et al., 2000) to optimize the Pareto front P f .
Revised: To solve this MOOP problem, we modify a well-known method named Elitist Non-Dominated sorting genetic algorithm (NSGA-II) (Deb et al., 2000) to optimize the Pareto front P f .

Revised: The main idea of NSGA-II is to rank the sampled architectures by non-dominated sorting and preserve a group of elite architectures.

Revised: Then a group of new architectures is sampled and trained by mutation of the current elite architectures on the P f .

Revised: The algorithm can be paralleled on multiple computation nodes and lift the P f simultaneously.

Revised: We modify the NSGA-II algorithm to become a NAS algorithm: a) To enable parallel searching on N computational nodes, we modify the non-dominated-sort method to generate exactly N mutated models for each generation, instead of a variable size as the original NSGA-II does.

Original:  Details of our NSGA-II can be found in Appendix B.2.2.
Revised: b) We deﬁne a group of mutation operations for our block/macro search space for NSGA-II to change the network structure dynamically. c) We add a parent computation node to measure the selected architecture’s training speed and generate the Pareto optimal models. Details of our NSGA-II can be found in Appendix B.2.2.

Original: Efﬁcient Training Model Zoo Z oo (ET-NAS).
Revised: Efﬁcient Training Model Zoo Z oo (ET-NAS).

Original: By the proposed NAS method, we then create an efﬁcient-training model zoo Z oo named ET-NAS which consists of K Pareto optimal models A ∗ i on
Revised: By the proposed NAS method, we then create an efﬁcient-training model zoo Z oo named ET-NAS which consists of K Pareto optimal models A ∗ i on

Original: P f .
Revised: P f .

Original: Then A ∗ i are pretrained by ImageNet.
Revised: Then A ∗ i are pretrained by ImageNet.

Original: Details of our NAS and A ∗ i can be found in Appendix B.
Revised: Details of our NAS and A ∗ i can be found in Appendix B.

Original:  O NLINE T ASK - ORIENTED F INE - TUNING S CHEDULE G ENERATION
Revised: 3. O NLINE T ASK - ORIENTED F INE - TUNING S CHEDULE G ENERATION

Original: With the help of efﬁcient training Z oo , the marginal computational cost of each user is minimized while they can enjoy the beneﬁt of NAS.
Revised: With the help of efﬁcient training Z oo , the marginal computational cost of each user is minimized while they can enjoy the beneﬁt of NAS.

Original: We then need to decide a suitable ﬁne-tuning schedule upon the user’s upcoming tasks.
Revised: We then need to decide a suitable ﬁne-tuning schedule upon the user’s upcoming tasks.

Original: Given user’s dataset D and ﬁne-tuning time constraint T l , an online regime generator G ( ., . ) is desired:
Revised: Given user’s dataset D and ﬁne-tuning time constraint T l , an online regime generator G ( ., . ) is desired:

Original: [Equation]
Revised: [Equation]

Original: val ) is maximized, (2) where the Regime FT includes all the hyperparameters required, i.e., lr schedule, total training steps, and frozen stages.
Revised: where the Regime FT includes all the hyperparameters required, i.e., lr schedule, total training steps, and frozen stages.

Original: G ( ., . ) also needs to pick up the most suitable pretrained model A ∗ i from Z oo .
Revised: G ( ., . ) also needs to pick up the most suitable pretrained model A ∗ i from Z oo .

Original: Note that existing search-based HPO methods require huge computational resources and cannot ﬁt in our online one-shot training scenario.
Revised: Note that existing search-based HPO methods require huge computational resources and cannot ﬁt in our online one-shot training scenario.

Original: Instead, we ﬁrst propose an online learning predictor Acc P to model the accuracy on the validation set Acc( A FTi , D val ) by the meta-data information.
Revised: Instead, we ﬁrst propose an online learning predictor Acc P to model the accuracy on the validation set Acc( A FTi , D val ) by the meta-data information.

Original: Then we can use the predictor to construct G ( ., . ) to generate an optimal hyperparameter setting and model.
Revised: Then we can use the predictor to construct G ( ., . ) to generate an optimal hyperparameter setting and model.

Original: 3.2. O NLINE L EARNING FOR M ODELING Acc( A FTi , D val )
Revised: 3.2. O NLINE L EARNING FOR M ODELING Acc( A FTi , D val )

Original: Recently, Li et al.
Revised: Recently, Li et al.

Original: (2020) suggest that the optimal hyperparameters for ﬁne-tuning are highly related to some data statistics such as domain similarity to the ImageNet.
Revised: (2020) suggest that the optimal hyperparameters for ﬁne-tuning are highly related to some data statistics such as domain similarity to the ImageNet.

Original: Thus, we hypothesis that we can model the ﬁnal accuracy by a group of predictors, e.g., model information, meta-data description, data statistics stat( D ) , domain similarity, and hyperparameters.
Revised: Thus, we hypothesis that we can model the ﬁnal accuracy by a group of predictors, e.g., model information, meta-data description, data statistics stat( D ) , domain similarity, and hyperparameters.

Original: We list the variables we considered to predict the accuracy result as follows:
Revised: We list the variables we considered to predict the accuracy result as follows:

Original: [Table]
Revised: [Figure]

Original: Those variables can be easily calculated ahead of the ﬁne-tuning.
Revised: Those variables can be easily calculated ahead of the ﬁne-tuning.

Original: One can prepare ofﬂine training data by ﬁne-tuning different kinds of dataset and collect the accuracy correspondingly and apply a
Revised: One can prepare ofﬂine training data by ﬁne-tuning different kinds of dataset and collect the accuracy correspondingly and apply a

Original: Multi-layer Perception Regression (MLP) ofﬂine on it.
Revised: Multi-layer Perceptron Regression (MLP) ofﬂine on it.

Original: However, online learning should be a more realistic setting for our problem.
Revised: However, online learning should be a more realistic setting for our problem.

Original: In cloud computing service or a GPU cluster, a sequence of ﬁnetuning requests with different data will arrive from time to time.
Revised: In cloud computing service or a GPU cluster, a sequence of ﬁnetuning requests with different data will arrive from time to time.

Original: The predictive model can be further improved by increasing the diversity of the data and the requests over time.
Revised: The predictive model can be further improved by increasing the diversity of the data and the requests over time.

Original: Using a ﬁxed depth of MLP model in the online setting may be problematic.
Revised: Using a ﬁxed depth of MLP model in the online setting may be problematic.

Original: Shallow networks maybe more preferred for small number of instances, while deeper model can achieve better performance when the sample size becomes larger.
Revised: Shallow networks maybe more preferred for small number of instances, while deeper model can achieve better performance when the sample size becomes larger.

Original: Inspired by Sahoo et al.
Revised: Inspired by Sahoo et al.

Original: (2017), we use an adaptive MLP regression to automatically adapt its model capacity from simple to complex over time.
Revised: (2017), we use an adaptive MLP regression to automatically adapt its model capacity from simple to complex over time.

Original: Given the input variables, the prediction of the accuracy is given by:
Revised: Given the input variables, the prediction of the accuracy is given by:

Revised: Acc P ( A

Original: [Equation]
Revised: [Equation]

Original:  , where l [Equation] [Equation] L
Revised: i , Regime FT , stat

Original: [Equation]
Revised: [Equation]

Original: f
Revised: D

Original: [Equation]
Revised: [Equation]

Original: =h
Revised: =

Original: [Equation]

Original: l
Revised: L

Original: [Equation]
Revised: [Equation]

Original: h [Equation] = RELU [Equation]
Revised: l = α l f l , where l = 0 , ..., L

Original: h

Original: [Equation]

Original: [Equation]

Original: h

Original: [Equation]
Revised: [Equation]

Original:  =[ A [Equation] i
Revised: 3) f l = h l W l , h l = RELU ( Φ l h l − 1 ) , h 0 =

Original: [Equation] Regime FT , stat( D )] .
Revised: [ A ∗ i , Regime FT , stat( D )] .

Original: The predicted accuracy is a weighted sum of the output f l of each intermediate fully-connected layer h l .
Revised: The predicted accuracy is a weighted sum of the output f l of each intermediate fully-connected layer h l .

Original: The W l and Φ l are the learnable weights of each fully-connected layer.
Revised: The W l and Φ l are the learnable weights of each fully-connected layer.

Original: The α l is a weight vector assigning the importance to each layer and (cid:107) α (cid:107)
Revised: The α l is a weight vector assigning the importance to each layer and (cid:107) α (cid:107)

Original: = 1 .
Revised: = 1 .

Original: Thus the predictor Acc P can automatically adapt its model capacity from simple to complex along with incoming tasks.
Revised: Thus the predictor Acc P can automatically adapt its model capacity from simple to complex along with incoming tasks.

Original: The learnable weight α l controls the importance of each intermediate layer and the ﬁnal predicted accuracy is a weighted sum of f l of them.
Revised: The learnable weight α l controls the importance of each intermediate layer and the ﬁnal predicted accuracy is a weighted sum of f l of them.

Original: The network can be updated by a Hedge Backpropagation (Freund & Schapire, 1999) in which α l is updated based on the loss suffered by this layer l as follows:
Revised: The network can be updated by a Hedge Backpropagation (Freund & Schapire, 1999) in which α l is updated based on the loss suffered by this layer l as follows:

Original: [Equation]
Revised: [Equation]

Original: where β ∈ (0 , 1) is the discount rate, the weight α (cid:48) l are re-normalized such that (cid:107) α (cid:107) = 1 , and η is the learning rate.
Revised: where β ∈ (0 , 1) is the discount rate, the weight α (cid:48) l are re-normalized such that (cid:107) α (cid:107) = 1 , and η is the learning rate.

Original: Thus, during the online update, the model can choose an appropriate depth by α l based on the performance of each output at that depth.
Revised: Thus, during the online update, the model can choose an appropriate depth by α l based on the performance of each output at that depth.

Original: By utilizing the online cumulative results, our generator gains experience that helps future prediction.
Revised: By utilizing the online cumulative results, our generator gains experience that helps future prediction.

Original: Generating Task-oriented Fine-tuning Schedule.
Revised: Generating Task-oriented Fine-tuning Schedule.

Original: Our schedule generator G then can make use of the performance predictor to ﬁnd the best training regime:
Revised: Our schedule generator G then can make use of the performance predictor to ﬁnd the best training regime:

Original: [Equation]
Revised: [Equation]

Original: Once the time constraint T l is provided, the max number of iterations for different A ∗ i can be calculated by an ofﬂine step-time lookup table for Z oo .
Revised: Once the time constraint T l is provided, the max number of iterations for different A ∗ i can be calculated by an ofﬂine step-time lookup table for Z oo .

Original: The corresponding meta-data variables can be then calculated for the incoming task.
Revised: The corresponding meta-data variables can be then calculated for the incoming task.

Original: The optimal selection of model and hyperparameters is obtained by ranking the predicted accuracy of all possible grid combinations.
Revised: The optimal selection of model and hyperparameters is obtained by ranking the predicted accuracy of all possible grid combinations.

Original: The Detailed algorithm can be found in the Appendix B.6.
Revised: The Detailed algorithm can be found in the Appendix B.6.

Revised: [Table]

Original:  E XPERIMENTAL R ESULTS 4. P RELIMINARY E XPERIMENTS
Revised:  E XPERIMENTAL R ESULTS 4. P RELIMINARY E XPERIMENTS

Original: We conduct a complete preliminary experiment to justify our motivation and model settings in the Appendix A.We ﬁrst verify some existing conclusions such as a) Fine-tuning performs better than training from scratch (Kornblith et al., 2019) so that our topic is very important for efﬁcient GPU training; b) learning rate and frozen stage are crucial for ﬁne-tuning (Guo et al., 2019), which needs careful adjustment during ﬁne-tuning.
Revised: We conduct a complete preliminary experiment to justify our motivation and model settings. Details can be found in the Appendix A.

Original: We further ﬁnd that for an efﬁcient ﬁne-tuning, the model matters most.
Revised: According to our experiments, we ﬁnd that for an efﬁcient ﬁnetuning, the model matters most.

Original: The suitable model should be selected according to the task and time constraints.
Revised: The suitable model should be selected according to the task and time constraints .

Original: Thus constructing a model zoo with various sizes of trainingefﬁcient models and picking up suitable models should be a good solution for faster ﬁne-tuning.
Revised: Thus constructing a model zoo with various sizes of training-efﬁcient models and picking up suitable models should be a good solution for faster ﬁne-tuning.

Original: [Table] [Table]
Revised: We also verify some existing conclusions: Fine-tuning performs better than training from scratch (Kornblith et al., 2019) so that our topic is very important for efﬁcient GPU training; Learning rate and frozen stage are crucial for ﬁne-tuning (Guo et al., 2019), which needs careful adjustment.

Original: 4. O FFLINE NAS AND M ODEL Z OO R ESULTS
Revised: 4. O FFLINE NAS AND M ODEL Z OO R ESULTS

Original:  During the NAS, we directly search on the ImageNet dataset(Russakovsky et al., 2015).
Revised: [Table] During the NAS, we directly search on the ImageNet dataset(Russakovsky et al., 2015).

Original: We ﬁrst search for a group of efﬁcient block structure, then use those block candidates to conduct the macro-level search.
Revised: We ﬁrst search for a group of efﬁcient block structure, then use those block candidates to conduct the macro-level search.

Original: We use a short training setting to evaluate each architecture.
Revised: We use a short training setting to evaluate each architecture.

Original: It takes about 1 hour on average for evaluating one architecture for the block-level search and 6 hours for the macrolevel search.
Revised: It takes about 1 hour on average for evaluating one architecture for the block-level search and 6 hours for the macro-level search.

Original: Paralleled on GPUs, it takes about one week on a 64-GPU cluster to conduct the whole search (5K+1K arch).
Revised: Paralleled on GPUs, it takes about one week on a 64-GPU cluster to conduct the whole search (5K+1K arch).

Original: Implementation details and intermediate results can be found in the Appendix B.
Revised: Implementation details and intermediate results can be found in the Appendix B.

Original: Faster Fine-tuning Model Zoo (ET-NAS).
Revised: Faster Fine-tuning Model Zoo (ET-NAS).

Original: After identifying the A ∗ i from our search, we fully train those models on ImageNet following common practice.
Revised: After identifying the A ∗ i from our search, we fully train those models on ImageNet following common practice.

Original: Note that all the models including ET-NAS-L can be easily pretrained on a regular 8-card GPU node since our model is trainingefﬁcient.
Revised: Note that all the models including ET-NAS-L can be easily pretrained on a regular 8-card GPU node since our model is training-efﬁcient.

Original: We are conﬁdent to release our models for the public to reproduce our results from scratch 2 and let the public to save their energy/CO2/cost .
Revised: We are conﬁdent to release our models for the public to reproduce our results from scratch 2 and let the public to save their energy/CO2/cost .

Original: Due to the length of the paper, we put the detailed encoding and architectures of the ﬁnal searched models in the Appendix B.4.
Revised: Due to the length of the paper, we put the detailed encoding and architectures of the ﬁnal searched models in the Appendix B.4.

Original: Surprisingly, we found that smaller models should use simpler structures of blocks while bigger models prefer complex blocks.
Revised: Surprisingly, we found that smaller models should use simpler structures of blocks while bigger models prefer complex blocks.

Original: Comparing our searched backbone to the conventional
Revised: Comparing our searched backbone to the conventional

Original: ResNet/ResNeXt, we ﬁnd that early stages in our models are very short which is more efﬁcient since feature maps in an early stage are very large and the computational cost is comparably large.
Revised: ResNet/ResNeXt, we ﬁnd that early stages in our models are very short which is more efﬁcient since feature maps in an early stage are very large and the computational cost is comparably large.

Original: This also veriﬁed our ﬁndings in Appendix B.4.1.
Revised: This also veriﬁed our ﬁndings in Appendix B.4.1.

Original: Comparison with the state-of-the-art ImageNet models.
Revised: Comparison with the state-of-the-art ImageNet models.

Original: We compare the training/inference efﬁciency of our searched ET-NAS with the SOTA ImageNet models such as MobileNetV3 (Howard et al., 2019), RegNet series (Radosavovic et al., 2020), and EfﬁcientNet series (Tan & Le, 2019) as shown in Table 2 and Figure 3.
Revised: We compare the training/inference efﬁciency of our searched ET-NAS with the SOTA ImageNet models such as MobileNetV3 (Howard et al., 2019), RegNet series (Radosavovic et al., 2020), and EfﬁcientNet series (Tan & Le, 2019) [Table] [Figure] [Figure]

Original:  Overall, our searched models outperform other SOTA ImageNet models in terms of training accuracy and training speed from Figure 3 (left).
Revised: [Figure] as shown in Table 2 and Figure 3. Overall, our searched models outperform other SOTA ImageNet models in terms of training accuracy and training speed from Figure 3 (left).

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: ET-NAS-G is about 6x training faster than RegNetY-16GF, and ET-NAS-I is about 1.5x training faster than EfﬁcientNetB3.
Revised: ET-NAS-G is about 6x training faster than RegNetY-16GF, and ET-NAS-I is about 1.5x training faster than EfﬁcientNetB3.

Original: Our models are also better than mobile setting models such as MobileNetV2/V3(Howard et al., 2019) and RegNetY-200MF.
Revised: Our models are also better than mobile setting models such as MobileNetV2/V3(Howard et al., 2019) and RegNetY-200MF.

Original: Although our model is optimized for fast training, we also compare the inference speed in Figure 3(right).
Revised: Although our model is optimized for fast training, we also compare the inference speed in Figure 3(right).

Original: Our models still have a very strong performance in terms of inference speed, outperforming RegNet series and achieving comparable performance with EfﬁcientNet.
Revised: Our models still have a very strong performance in terms of inference speed, outperforming RegNet series and achieving comparable performance with EfﬁcientNet.

Original: In Figure 3 (mid), our method is more training efﬁcient than other
Revised: In Figure 3 (mid), our method is more training efﬁcient than other

Original: NAS results, e.g.some evolution-based NAS methods such as AmoebaNet, OFANet.
Revised: NAS results, e.g.some evolution-based NAS methods such as AmoebaNet, OFANet.

Revised: [Figure]

Original: What makes our network training efﬁcient?
Revised: What makes our network training efﬁcient?

Original: To answer this, we deﬁne an efﬁciency score and conduct a statistical analysis of different factors for efﬁcient-training (Details can be found in Appendix B.4.1).
Revised: To answer this, we deﬁne an efﬁciency score and conduct a statistical analysis of different factors for efﬁcient-training (Details can be found in Appendix B.4.1).

Original: We have the following conclusions: a)
Revised: We have the following conclusions: a)

Original: By observing optimal A ∗ i , smaller models should use simpler blocks while bigger models prefer complex blocks.
Revised: By observing optimal A ∗ i , smaller models should use simpler blocks while bigger models prefer complex blocks.

Original: Using the same block structure for all sizes of models (Tan & Le, 2019; Radosavovic et al., 2020) may not be optimal.b) Adding re- [Figure]
Revised: Using the same block structure for all sizes of models (Tan & Le, 2019; Radosavovic et al., 2020) may not be optimal.

Original: [Figure] [Figure] dundant skip-connections which have great memory access cost will decrease the training efﬁciency of the model thus existing topological cell-level search space such as DARTS (Liu et al., 2018b), AmoebaNet (Real et al., 2019a), and NASBench101 (Dong & Yang, 2019) is not efﬁcient.
Revised: b) Adding redundant skip-connections which have great memory access cost will decrease the training efﬁciency of the model thus existing topological cell-level search space such as DARTS (Liu et al., 2018b), AmoebaNet (Real et al., 2019a), and NASBench101 (Dong & Yang, 2019) is not efﬁcient.

Original: c)
Revised: c)

Original: The computation allocation on different stages is crucially important.
Revised: The computation allocation on different stages is crucially important.

Original: Simply increasing depth/width to expand the model as Tan & Le (2019) may not be optimal will downgrade the performance.
Revised: Simply increasing depth/width to expand the model as Tan & Le (2019) may not be optimal will downgrade the performance.

Original: To conclude our novel joint search space contributes most to the training efﬁciency.
Revised: To conclude our novel joint search space contributes most to the training efﬁciency.

Original: 4. R ESULTS FOR O NLINE A DAPTIVE P REDICTOR Acc P
Revised: 4. R ESULTS FOR O NLINE A DAPTIVE P REDICTOR Acc P

Original: Experimental Settings .
Revised: Experimental Settings .

Original: We evaluate our online algorithm based on ten widely used image classiﬁcation datasets, that cover various ﬁne-tuning tasks as shown in Table 1.
Revised: We evaluate our online algorithm based on ten widely used image classiﬁcation datasets, that cover various ﬁne-tuning tasks as shown in Table 1.

Original: Five of them (in bold) are chosen to be the online learning training set (meta-training dataset).
Revised: Five of them (in bold) are chosen to be the online learning training set (meta-training dataset).

Original: 30K samples are collected by continually sampling a subset of each dataset and ﬁne-tuning with a randomized hyperparameters on it.
Revised: 30K samples are collected by continually sampling a subset of each dataset and ﬁne-tuning with a randomized hyperparameters on it.

Original: Each subset varies from #. classes and #. images.
Revised: Each subset varies from #. classes and #. images.

Original: The variables in Section 3.2.1 are calculated accordingly.
Revised: The variables in Section 3.2.1 are calculated accordingly.

Original: The ﬁne-tuning accuracy is then evaluated on the test set.
Revised: The ﬁne-tuning accuracy is then evaluated on the test set.

Original: Then 30K sample is split into 24K meta-training samples and 6K meta-validation samples.
Revised: Then 30K sample is split into 24K meta-training samples and 6K meta-validation samples.

Original: Ouradaptive MLP with hedge backpropagation is better in the online setting of predicting the ﬁne-tuning
Revised: Our

Original: [Table]
Revised: [Table]

Original: Then an adaptive MLP regression in Eq 3 are used to ﬁt the data and predict the
Revised: Then an adaptive MLP regression in Eq 3 are used to ﬁt the data and predict the

Original: Acc ( A FTi , D val ) .
Revised: Acc ( A FTi , D val ) .

Original: We use L = 10 withunits in each hidden layer.
Revised: We use L = 10 withunits in each hidden layer.

Original: We use a learning rate of 0 . 01 and β = 0 . 99 .
Revised: We use a learning rate of 0 . 01 and β = 0 . 99 .

Original: As baselines, we also compare the results of using ﬁxed MLP with plain backpropagation with different layers( L = 3 , 6 , 10 , 14 ).
Revised: As baselines, we also compare the results of using ﬁxed MLP with plain backpropagation with different layers( L = 3 , 6 , 10 , 14 ).

Original: MAE (mean absolute error) and MSE (mean square error) are performance metrics to measure the cumulative error with different segments of the tasks stream.
Revised: MAE (mean absolute error) and MSE (mean square error) are performance metrics to measure the cumulative error with different segments of the tasks stream.

Original: Comparison of online learning method.
Revised: Comparison of online learning method.

Original: The cumulative error obtained by all the baselines and the proposed method to predict the ﬁne-tuning accuracy is shown in Table 4.
Revised: The cumulative error obtained by all the baselines and the proposed method to predict the ﬁne-tuning accuracy is shown in Table 4.

Original: It can be seen that our adaptive MLP with hedge backpropagation is better than ﬁxed MLP in terms of the cumulative error of the predicted accuracy.
Revised: It can be seen that our adaptive MLP with hedge backpropagation is better than ﬁxed MLP in terms of the cumulative error of the predicted accuracy.

Original: Our method enjoys the beneﬁt from the adaptive depth which allows faster convergence in the initial stage and strong predictive power in the later stage.
Revised: Our method enjoys the beneﬁt from the adaptive depth which allows faster convergence in the initial stage and strong predictive power in the later stage.

Original: 4. F INAL NASOA R ESULTS
Revised: 4. F INAL NASOA R ESULTS

Revised: [Figure]

Revised: ﬁxed MLP predictor.

Revised: “Our NASOA” is the our whole pipeline with both training efﬁcient model zoo and online adaptive scheduler.

Revised: Without additional search cost (x40), NASOA can reach similar performance of BOHB.

Revised: [Table]

Original: To evaluate the performance of our whole NASOA, we select four time constraints on the testing datasets and use Acc P ( . ) to test the ﬁne-tuning accuracy.
Revised: To evaluate the performance of our whole NASOA, we select four time constraints on the testing datasets and use Acc P ( . ) to test the ﬁne-tuning accuracy.

Original: The testing datasets are MIT67, Food101,
Revised: The testing datasets are MIT67, Food101,

Original: Aircrafts, Blood-cell, and Stanford-Car.
Revised: Aircrafts, Blood-cell, and Stanford-Car.

Original: The shortest/longest time constraint are the time of ﬁnetuning 10/50 epochs for ResNet18/ResNet101.
Revised: The shortest/longest time constraint are the time of ﬁne-tuning 10/50 epochs for ResNet18/ResNet101.

Original: The rest are equally divided into the log-space.
Revised: The rest are equally divided into the log-space.

Original: For our NASOA, we generate the ﬁne-tuning schedules by maximizing the predicted accuracy in Eq 2.
Revised: For our NASOA, we generate the ﬁne-tuning schedules by maximizing the predicted accuracy in Eq 2.

Original: T o p - 1 A cc u r a c y

Original: Stanford-Car 0.55 0.

Original: MIT 0.25 0.35 0.

Original: FGVC Aircraft 0.85 0.855 0.

Original: Blood-cell 0.8 0.

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure] [Table] We also conduct ﬁne-tuning on various candidates of baselines such as ResNet (R18 to R152), RegNet (200MF to 16GF), and EfﬁcientNet (B0 to B3) with the default hyperparameter setting in Li et al.
Revised: We also conduct ﬁnetuning on various candidates of baselines such as ResNet (R18 to R152), RegNet (200MF to 16GF), and EfﬁcientNet (B to B3) with the default hyperparameter setting in Li et al.

Original: Comparison of the ﬁnal ﬁne-tuning results with the SOTA networks.
Revised: Comparison of the ﬁnal ﬁne-tuning results with the SOTA networks.

Original: We plot the time versus accuracy comparison in Figure 4.
Revised: We plot the time versus accuracy comparison in Figure 5.

Original: As can be seen, the model and training regime generated by our NASOA can outperform the upper bound of other methods in most cases.
Revised: As can be seen, the model and training regime generated by our NASOA can outperform the upper bound of other methods in most cases.

Original: On average, our methods can improve around 2.1%/7.4% accuracy than the best model of RegNet/EfﬁcientNet series under various time constraints and tasks.
Revised: On average, our methods can improve around 2.1%/7.4% accuracy than the best model of RegNet/EfﬁcientNet series under various time constraints and tasks.

Original: It is noteworthy that
Revised: It is noteworthy that

Original: our
Revised: our

Original: NASOA performs better especially in the case of short time constraint, which demonstrates that our schedule generator is capable provide both efﬁcient and effective regimes for fast ﬁne-tuning.
Revised: NASOA performs better especially in the case of short time constraint, which demonstrates that our schedule generator is capable provide both efﬁcient and effective regimes for fast ﬁne-tuning.

Original: [Table]
Revised: [Table]

Original: Comparison of the ﬁnal ﬁne-tuning results with the HPO methods.
Revised: with the HPO methods.

Original: In Table 4, we compare our method with the HPO methods which optimizing the hyperparameters and picking up models in ResNet, RegNetY, and EfﬁcientNet series.
Revised: In Table 4, we compare our method with the HPO methods which optimizing the hyperparameters and picking up models in ResNet, RegNetY, and EfﬁcientNet series.

Original: “HPO only” means the method only optimizes the hyperparameters with a ﬁxed model
Revised: “HPO only” means the method only optimizes the hyperparameters with a ﬁxed model

Original: RegNetY-16GF.
Revised: RegNetY-16GF.“OA only” is our online schedule generator with RegNet series models. “Our Z oo ” means using our ET models zoo to ﬁnd suitable model. “Fixed MLP Predictor” is the ofﬂine baseline with ﬁxed MLP predictor (L=10) with our model zoo.

Revised: “Our NASOA” is the our whole pipeline with both training efﬁcient model zoo and online adaptive scheduler.

Revised: Comparing to the ofﬂine baseline with our NASOA, our online adaption module can boost the average performance by 2.17%.

Original: It can  be found that our method can save up to 40x computational cost compared to HPO methods while reaching similar performance.
Revised: It can also be found that our method can save up to 40x computational cost compared to HPO methods while reaching similar performance.

Original: With more computational budget, our model zoo with BOHB search can reach even higher accuracy (+avg. 10.28%).
Revised: With more computational budget, our model zoo with BOHB search can reach even higher accuracy (+avg. 10.28%).

Original: Ablative interpretation of performance superiority.
Revised: Ablative interpretation of performance superiority.

Original: Table 5 calculates the average ﬁne-tuning accuracy over tasks.
Revised: Table 5 calculates the average ﬁne-tuning accuracy over tasks.

Original: Our NAS model zoo can greatly increase the ﬁne-tuning average accuracy from 77 .
Revised: Our NAS model zoo can greatly increase the ﬁne-tuning average accuracy from 77 .

Original: 17 % to 87 .
Revised: 17 % to 87 .

Original: 45 %, which is the main contribution of the performance superiority.
Revised: 45 %, which is the main contribution of the performance superiority.

Original: Using our online adaptive scheduler instead of BOHB can signiﬁcantly reduce the computational cost (-40x).
Revised: Using our online adaptive scheduler instead of BOHB can signiﬁcantly reduce the computational cost (-40x).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose the ﬁrst efﬁcient task-oriented ﬁne-tuning framework aiming at saving the resources for GPU clusters and cloud computing.
Revised: We propose the ﬁrst efﬁcient task-oriented ﬁne-tuning framework aiming at saving the resources for GPU clusters and cloud computing.

Original: The joint NAS and online adaption strategy achieve much better ﬁne-tuning results in terms of both accuracy and speed.
Revised: The joint NAS and online adaption strategy achieve much better ﬁne-tuning results in terms of both accuracy and speed.

Original: The searched architectures are more training-efﬁcient than very strong baselines such as RegNet and EfﬁcientNet.
Revised: The searched architectures are more training-efﬁcient than very strong baselines such as RegNet and EfﬁcientNet.

Original: Our experiments on multiple datasets show our NASOA achieves 40x speed-up comparing to BOHB.
Revised: Our experiments on multiple datasets show our NASOA achieves 40x speed-up comparing to BOHB.

Original: The proposed
Revised: The proposed

Original: NASOA can be well adapted to more tasks such as detection and segmentation in the future.
Revised: NASOA can be well adapted to more tasks such as detection and segmentation in the future.

Original: A P RELIMINARY EXPERIMENTS A. E XPERIMENTS S ETTINGS
Revised: A P RELIMINARY EXPERIMENTS A. E XPERIMENTS S ETTINGS

Original: The preliminary experiments aim at ﬁguring out what kinds of factors impact the speed and accuracy of ﬁne-tuning.
Revised: The preliminary experiments aim at ﬁguring out what kinds of factors impact the speed and accuracy of ﬁne-tuning.

Original: We ﬁne-tune several ImageNet pretrained backbones on various datasets as shown in Table 6 (right) and exam different settings of hyperparameters by a grid search such as: learning rate (0.0001, 0.001, 0.01, 0.1), frozen stages (-1,0,1,2,3), and frozen BN (-1,0,1,2,3).
Revised: We ﬁne-tune several ImageNet pretrained backbones on various datasets as shown in Table 6 (right) and exam different settings of hyperparameters by a grid search such as: learning rate (0.0001, 0.001, 0.01, 0.1), frozen stages (-1,0,1,2,3), and frozen BN (-1,0,1,2,3).

Original: Frozen stages/frozen BN = k means 1 to k th stage’s parameters/BN statistics are not updated during training.
Revised: Frozen stages/frozen BN = k means 1 to k th stage’s parameters/BN statistics are not updated during training.

Original: The training settings most follow Li et al.
Revised: The training settings most follow Li et al.

Original: and we report the Top-1 validation accuracy and training time.
Revised: and we report the Top-1 validation accuracy and training time.

Original: Its detailed experiment settings are hyperparameters are listed as follows:
Revised: Its detailed experiment settings are hyperparameters are listed as follows:

Original: [Table]

Original: Comparing ﬁne-tuning and training from scratch .
Revised: Comparing ﬁne-tuning and training from scratch .

Original: We use ResNet series (R-18 to R-50) to evaluate the effect of ﬁne-tuning and training from scratch.
Revised: We use ResNet series (R-18 to R-50) to evaluate the effect of ﬁne-tuning and training from scratch.

Original: Following Li et al.
Revised: Following Li et al.

Original: , we train networks on Flowers102, CUB-Birds, MIT67 and Caltech101 datasets for 600 epochs for training from scratch and 350 epochs for ﬁne-tuning to ensure all models converge on all datasets.
Revised: , we train networks on Flowers102, CUB-Birds, MIT67 and Caltech101 datasets for 600 epochs for training from scratch and 350 epochs for ﬁne-tuning to ensure all models converge on all datasets.

Original: We use SGD optimizer with an initial learning rate 0.01, weight decay 1e-4, momentum 0.9.
Revised: We use SGD optimizer with an initial learning rate 0.01, weight decay 1e-4, momentum 0.9.

Original: The learning rate is decreased by factor 10 at 400 and 550 epoch for training from scratch and 150, 250 epoch for ﬁne-tuning.
Revised: The learning rate is decreased by factor 10 at 400 and 550 epoch for training from scratch and 150, 250 epoch for ﬁne-tuning.

Original: Optimal learning rate and frozen stage .
Revised: Optimal learning rate and frozen stage .

Original: We perform a simple grid search on Flowers102,
Revised: We perform a simple grid search on Flowers102,

Original: Stanford-Car, CUB-Birds, MIT67, Stanford-Dog, and Caltech101 datasets with ResNet50 to ﬁnd optimal learning rate andfrozen stage on different datasets with the default ﬁne-tune setting in Li et al.
Revised: Stanford-Car, CUB-Birds, MIT67, Stanford-Dog, and Caltech101 datasets with ResNet50 to ﬁnd optimal learning rate and

Original: The hyperparameters ranges are: learning rate (0.1, 0.01, 0.001, 0.0001), frozen stage (-1, 0, 1, 2, 3).

Original: Comparing different frozen stages and networks along time .
Revised: Comparing different frozen stages and networks along time .

Original: We ﬁx different stages of ResNet to analyze the inﬂuence of different frozen stages to the accuracy and along the training time on
Revised: We ﬁx different stages of ResNet to analyze the inﬂuence of different frozen stages to the accuracy and along the training time on

Original: Flowers102, Stanford-Car, CUB-Birds, MIT67, Stanford-Dog, and Caltech101 datasets.
Revised: Flowers102, Stanford-Car, CUB-Birds, MIT67, Stanford-Dog, and Caltech101 datasets.

Original: We pick  the training curves on CUB-Birds and Caltech101 to in the main text of this paper.
Revised: We pick [Table] [Table] the training curves on CUB-Birds and Caltech101 to in the main text of this paper.

Original: We also compare the ﬁne-tune results along time with various networks on these datasets as shown in Figure 6.
Revised: We also compare the ﬁne-tune results along time with various networks on these datasets as shown in Figure 7.

Original: On Caltech101, ResNet50 dominates the training curve at the very beginning.
Revised: On Caltech101, ResNet50 dominates the training curve at the very beginning.

Original: However, on other datasets, ResNet18 and ResNet34 can perform better then ResNet50 when the training time is short.
Revised: However, on other datasets, ResNet18 and ResNet34 can perform better then ResNet50 when the training time is short.

Original: A. F INDINGS OF THE PRELIMINARY EXPERIMENTS
Revised: A. F INDINGS OF THE PRELIMINARY EXPERIMENTS

Original: With those preliminary experiments, we summarize our ﬁndings as follows.
Revised: With those preliminary experiments, we summarize our ﬁndings as follows.

Original: Some of the ﬁndings are also veriﬁed by some existing works.
Revised: Some of the ﬁndings are also veriﬁed by some existing works.

Original: [Table] [Figure] [Figure] • Fine-tuning performs always better than training from scratch.
Revised: • Fine-tuning performs always better than training from scratch.

Original: As shown in Table 6, ﬁnetuning shows superior results than training from scratch in terms of both accuracy and training time for all the datasets.
Revised: As shown in Table 6, ﬁnetuning shows superior results than training from scratch in terms of both accuracy and training time for all the datasets.

Original: This ﬁnding is also veriﬁed by Kornblith et al.
Revised: This ﬁnding is also veriﬁed by Kornblith et al.

Original: Thus, ﬁne-tuning is the most common way to train a new dataset and our framework can be generalized to applications. 
Revised: Thus, ﬁne-tuning is the most common way to train a new dataset and our framework can be generalized to applications. [Table] [Table] [Figure]

Original: • We should the optimize learning rate and frozen stage for each dataset.
Revised: T o p - 1 A cc u r a c y Training Time Stanford-Car architecture acc v.s. time 0.55 0. 100 Training time

Revised: CUB-Birds architecture accv.s.

Revised: time 0.65 0.67 0.

Revised: [Figure]

Revised: v.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original:  From Table 7, it seems that the optimal learning rate and optimal frozen stage found by grid search are different for various datasets.
Revised: [Figure] res18 res res • We should the optimize learning rate and frozen stage for each dataset. From Table 7, it seems that the optimal learning rate and optimal frozen stage found by grid search are different for various datasets.

Original: Figure 5also shows that the number of the frozen stages will affect both training time and ﬁnal accuracy.
Revised: Figure 6also shows that the number of the frozen stages will affect both training time and ﬁnal accuracy.

Original: Guo et al.
Revised: Guo et al.

Original: (2019) also showed that frozen different stages are crucial for ﬁne-tuning task.
Revised: (2019) also showed that frozen different stages are crucial for ﬁne-tuning task.

Original: Those two hyperparameters should be optimized for different datasets.
Revised: Those two hyperparameters should be optimized for different datasets.

Original: • Model matters most.
Revised: • Model matters most.

Original: Suitable model should be selected according to the task and time constraints.
Revised: Suitable model should be selected according to the task and time constraints.

Original: Figure 5 (right) suggests that always choosing the biggest model to ﬁne-tune may not be an optimal choice, smaller model can be better than the bigger model if the training time is limited.
Revised: Figure 6 (right) suggests that always choosing the biggest model to ﬁne-tune may not be an optimal choice, smaller model can be better than the bigger model if the training time is limited.

Original: On the other hand, it is also important to consider the training efﬁciency of the model since a better model can be converged faster by a limited GPU budget.
Revised: On the other hand, it is also important to consider the training efﬁciency of the model since a better model can be converged faster by a limited GPU budget.

Original: For example, Figureshows that if the time constraint is short, we should choose a smaller network i.e. ResNet18 here.
Revised: For example, Figureshows that if the time constraint is short, we should choose a smaller network i.e. ResNet18 here.

Original: Thus, it is urgent to construct a training-efﬁcient model zoo.
Revised: Thus, it is urgent to construct a training-efﬁcient model zoo.

Original: • BN running statistics should not be frozen during ﬁne-tuning.
Revised: • BN running statistics should not be frozen during ﬁne-tuning.

Original: We found that frozen BN has a very limited effect on the training time (less than ± 5% ), while not freezing BN will lead to better results in almost all the datasets.
Revised: We found that frozen BN has a very limited effect on the training time (less than ± 5% ), while not freezing BN will lead to better results in almost all the datasets.

Original: Thus, BN is not frozen all experiments for our NASOA.
Revised: Thus, BN is not frozen all experiments for our NASOA.

Original: B D ETAILS OF THE O FFLINE NAS B. S EARCH S PACE E NCODINGS
Revised: B D ETAILS OF THE O FFLINE NAS B. S EARCH S PACE E NCODINGS

Original: The search space of our architectures is composed of block and macro levels, where the former decides what a block is composed of, such as operators, number of channels, and skip connections, while the latter is concerned about how to combine the block into a whole network, e.g., when to do down-sampling, and where to change the number of channels.
Revised: The search space of our architectures is composed of block and macro levels, where the former decides what a block is composed of, such as operators, number of channels, and skip connections, while the latter is concerned about how to combine the block into a whole network, e.g., when to do down-sampling, and where to change the number of channels.

Original: B.1. B LOCK - LEVEL ARCHITECTURE
Revised: B.1. B LOCK - LEVEL ARCHITECTURE

Original: Block-level design.
Revised: Block-level design.

Original: A block consists of at most three operators, each of which is divided intospecies and has 5 different number of output channels.
Revised: A block consists of at most three operators, each of which is divided into 5 species and has 5 different number of output channels.

Original: Each kind of operator is denoted by an op number, and the output channel of the operator is decided by the ratio between it and that of the current block.
Revised: Each kind of operator is denoted by an op number, and the output channel of the operator is decided by the ratio between it and that of the current block.

Original: Details are shown in Table B.1.1.
Revised: Details are shown in Table B.1.1.

Original: By default, there is a skip connection between the input and output of the block, which sums their values up.
Revised: By default, there is a skip connection between the input and output of the block, which sums their values up.

Original: In addition to that, at most 3 other skip connections are contained in a block, which either adds or concatenates the values between them.
Revised: In addition to that, at most 3 other skip connections are contained in a block, which either adds or concatenates the values between them.

Original: Each operation is followed by a batch normalization layer, and after all the skip connections are calculated, a ReLU layer is triggered.a
Revised: Each operation is followed by a batch normalization layer, and after all the skip connections are calculated, a ReLU layer is triggered.

Original: Block-level encoding.
Revised: Block-level encoding.

Original: The encoding of each block-level architecture is composed of two parts separated by ‘-’, which considers the operators and skip connections respectively.
Revised: The encoding of each block-level architecture is composed of two parts separated by ‘-’, which considers the operators and skip connections respectively.

Original: For the ﬁrst part (operators part), each operator is represented by two numbers: op number and ratio number (shown in Table B.1.1).
Revised: For the ﬁrst part (operators part), each operator is represented by two numbers: op number and ratio number (shown in Table B.1.1).

Original: As the output channel of the last operation always equals to that of the current block, the ratio number of this operator is removed.
Revised: As the output channel of the last operation always equals to that of the current block, the ratio number of this operator is removed.

Original: Therefore, the encoding of a block with n operators always has length 2 n − 1 for the ﬁrst part of block-level encoding.
Revised: Therefore, the encoding of a block with n operators always has length 2 n − 1 for the ﬁrst part of block-level encoding.

Original: For the second part (skip connections part), every skip connection consists of one letter for addition (‘a’) / concatenation (‘c’), and two numbers for place indices.
Revised: For the second part (skip connections part), every skip connection consists of one letter for addition (‘a’) / concatenation (‘c’), and two numbers for place indices.

Original: n operators separate the block to n +1 parts, which are indexed with 0 , 1 , 2 , . . . , n .
Revised: n operators separate the block to n + parts, which are indexed with 0 , 1 , 2 , . . . , n .

Original: Thus ‘a01’ means summing up the value before and after the ﬁrst operator.
Revised: Thus ‘a01’ means summing up the value before and after the ﬁrst operator.

Original: Since the skip connection between the beginning and end of the block always exists, it is not shown in the encoding.
Revised: Since the skip connection between the beginning and end of the block always exists, it is not shown in the encoding.

Original: Thus this part has length 3 k − 3 (possibly 0 ) when there is k skip connections.
Revised: Thus this part has length 3 k − 3 (possibly 0 ) when there is k skip connections.

Original: Some of the encoding examples are shown in Figure 7.
Revised: Some of the encoding examples are shown in Figure 8.

Original: B.1. M ACRO - LEVEL ARCHITECTURE
Revised: B.1. M ACRO - LEVEL ARCHITECTURE

Original: Macro-level design.
Revised: Macro-level design.

Original: We only consider networks with exactly 4 stages.
Revised: We only consider networks with exactly 4 stages.

Original: The ﬁrst block of each stage (except Stage 1) reduces the resolution of both width and height by half, where the stride 2 is added to the ﬁrst operator that is not conv1x1.
Revised: The ﬁrst block of each stage (except Stage 1) reduces the resolution of both width and height by half, where the stride 2 is added to the ﬁrst operator that is not conv1x1.

Original: Other blocks do not change the resolution.
Revised: Other blocks do not change the resolution.

Original: One block’s output channel is either the same, or an integer multiple of the input channel.
Revised: One block’s output channel is either the same, or an integer multiple of the input channel.

Original: Macro-level encoding.
Revised: Macro-level encoding.

Original: The 4 stages are divided apart by 3 ‘-’ signs.
Revised: The 4 stages are divided apart by 3 ‘-’ signs.

Original: For every stage, each block is represented by an integer, which shows the ratio between output and input channel for this block.
Revised: For every stage, each block is represented by an integer, which shows the ratio between output and input channel for this block.

Original: B.1. E NCODING AS A WHOLE
Revised: B.1. E NCODING AS A WHOLE

Original: Thus the whole backbone can be encoded by simply concatenating the block and macro encoding.
Revised: Thus the whole backbone can be encoded by simply concatenating the block and macro encoding.

Original: The encoding of the whole network is formatted as:
Revised: The encoding of the whole network is formatted as:

Original: [Equation]
Revised: [Equation]

Original: Some common architectures, including ResNet and Wide ResNet can be accurately represented by our encoding scheme, which is shown in Table B.1.3.
Revised: Some common architectures, including ResNet and Wide ResNet can be accurately represented by our encoding scheme, which is shown in Table B.1.3.

Original: [Table] [Table] N ON - DOMINATED S ORTING A LGORITHM
Revised: B. NAS S EARCH A LGORITHM B.2. N ON - DOMINATED S ORTING A LGORITHM

Original: Non-dominated sorting is mainly used to sort the solutions in population according to the Pareto dominance principle , which plays a very important role in the selection operation of many multiobjective evolutionary algorithms.
Revised: Non-dominated sorting is mainly used to sort the solutions in population according to the Pareto dominance principle , which plays a very important role in the selection operation of many multiobjective evolutionary algorithms.

Original: In non-dominated sorting, an individual A is said to dominate another individual B, if and only if there is no objective of A worse than that objective of B and there is at least one objective of A better than that objective of B. Without loss of generality, we assume that the solutions of a population S can be assigned to K Pareto fronts F i , i = 1 , 2 , . . .
Revised: In non-dominated sorting, an individual A is said to dominate another individual B, if and only if there is no objective of A worse than that objective of B and there is at least one objective of A better than that objective of B. Without loss of generality, we assume that the solutions of a population S can be assigned to K Pareto fronts F i , i = 1 , 2 , . . .

Original: , K .
Revised: , K .

Original: Non-dominated sorting ﬁrst selects all the non-dominated solutions from population S and assigns them to F 1 (the rank 1 front)
Revised: Non-dominated sorting ﬁrst selects all the non-dominated solutions from population S and assigns them to F 1 (the rank 1 front)

Original: ; it then selects all the non-dominated solutions from the remaining solutions and assigns them to F 2 (the rank 2 front); it repeats the above process until all individuals have been assigned to a Pareto front .B.  [Table]
Revised: ; it then selects all the non-dominated solutions from the remaining solutions and assigns them to F 2 (the rank 2 front); it repeats the above process until all individuals have been assigned to a Pareto front .

Original:  NSGA-II: E LITIST N ON -D OMINATED S ORTING G ENETIC A LGORITHM
Revised: B.2. NSGA-II: E LITIST N ON -D OMINATED S ORTING G ENETIC A LGORITHM

Original: To solve the problem in Eq. 1, Elitist Non-Dominated sorting genetic algorithm (NSGA-II) (Deb et al., 2000) is adopted to optimize the Pareto front P f as shown in Algorithm 1. 
Revised: To solve the problem in Eq. 1, Elitist Non-Dominated sorting genetic algorithm (NSGA-II) (Deb et al., 2000) is adopted to optimize the Pareto front P f as shown in Algorithm 1. In this paper, [Table] =

Original: In this paper, we choose this kind of sample-based NAS algorithm instead of many popular parameter-sharing
Revised: [Table] [Table] [Table] , group=  ×

Revised: [Table]

Revised: [Table]

Revised: =

Revised: c ×

Revised: [Table]

Revised: we choose this kind of sample-based NAS algorithm instead of many popular parameter-sharing

Original: NAS method.
Revised: NAS method.

Original: This is because we want to further analysis of the sampled architectures and achieve insights and conclusions of the efﬁcient training.
Revised: This is because we want to further analysis of the sampled architectures and achieve insights and conclusions of the efﬁcient training.

Original: The main idea of NSGA-II is to rank the sampled architectures by non-dominated sorting and preserve a group of elite architectures.
Revised: The main idea of NSGA-II is to rank the sampled architectures by non-dominated sorting and preserve a group of elite architectures.

Original: Then a group of new architectures is sampled and trained by mutation of the current elite architectures on the P f .
Revised: Then a group of new architectures is sampled and trained by mutation of the current elite architectures on the P f .

Original: The algorithm can be paralleled on multiple computation nodes and lift the P f simultaneously.
Revised: The algorithm can be paralleled on multiple computation nodes and lift the P f simultaneously.

Original: The mutation in the block-level search space includes adding new skip-connection, modifying the current operations and ratios.
Revised: The mutation in the block-level search space includes adding new skip-connection, modifying the current operations and ratios.

Original: Meanwhile, the mutation in the macro-level search space includes randomly adding or deleting one block in one stage, exchanging the position of doubling channel block with its neighbor, and modifying the base channels.
Revised: Meanwhile, the mutation in the macro-level search space includes randomly adding or deleting one block in one stage, exchanging the position of doubling channel block with its neighbor, and modifying the base channels.

Original: This well-known NSGA-II is easy to implement and we can easily monitor the improvement of each iteration.
Revised: This well-known NSGA-II is easy to implement and we can easily monitor the improvement of each iteration.

Original: The stop criterion depends on the time limit or the computation cost constraints.
Revised: The stop criterion depends on the time limit or the computation cost constraints.

Original: B. NAS I MPLEMENTATION D ETAILS B.3. B LOCK - LEVEL SEARCH
Revised: 3: Evaluate P t 4: while not stop criterion do 5: Apply non-dominated sorting to P t to obtain non-dominated fronts A ∗ i 6: Sort A ∗ i by Crowding distance and left top-N A ∗ j as Parents 7: Create new generation Q t by mutation on current A ∗ j 8:

Revised: Train the Q t on N computation nodes and Evaluate the accuracy of Q t .

Revised: P t +1 ← Q t ∪ P t 10: t = t +11: end while

Revised: B. NAS I MPLEMENTATION D ETAILS B.3. B LOCK - LEVEL SEARCH

Original: In the phase of block-level search, a proxy task of ImageNet is created, which is a subset sampled fromits training set.
Revised: In the phase of block-level search, a proxy task of ImageNet is created, which is a subset sampled fromits training set.

Original: This subset constitutes 100 labels, each of which has 500 images as the training set, and 100 as the validation set.
Revised: This subset constitutes 100 labels, each of which has 500 images as the training set, and 100 as the validation set.

Original: We call this dataset ImageNet-100 in the following parts of this paper.
Revised: We call this dataset ImageNet-100 in the following parts of this paper.

Original: To avoid interference with macro architecture, the macro-level architecture is ﬁxed to be the same as that of ResNet50.
Revised: To avoid interference with macro architecture, the macro-level architecture is ﬁxed to be the same as that of ResNet50.

Original: Each model is trained by ImageNet-100 with a batch size of 32 for 90 epochs and learning rate 0.1, which takes 3˜10 hours on a single NVIDIA Tesla-V100 GPU.
Revised: Each model is trained by ImageNet-100 with a batch size of 32 for 90 epochs and learning rate 0.1, which takes 3˜10 hours on a single NVIDIA Tesla-V100 GPU.

Original: We do a random search at ﬁrst, which uniformly samples all the valid blocks in the search space.
Revised: We do a random search at ﬁrst, which uniformly samples all the valid blocks in the search space.

Original: Evolutionary
Revised: Evolutionary

Original: Algorithm (EA) is then performed with three kinds of mutations: 1) replace one operator with Input Stop criterion, Search Space, number of computation nodes N. 1: t =2: P t ← Random ( A ) , generate a group of initial architectures.
Revised: Algorithm (EA) is then performed with three kinds of mutations: 1) replace one operator with another; 2) change the output channel of one layer; 3) Add/remove/modify a skip connection.

Original: 3: Evaluate P t 4: while not stop criterion do 5: Apply non-dominated sorting to P t to obtain non-dominated fronts A ∗ i 6:
Revised: We keep updating the Pareto Front between step time and accuracy during the whole process.

Original: Sort A ∗ i by Crowding distance and left top-N A ∗ j as Parents 7:
Revised: As a [Table] ET-NAS-J 211- 64 411-2111-21111111111111111111111103.

Original: Create new generation Q t by mutation on current A ∗ j 8:
Revised: ET-NAS-K 02031-a02 64 112187.

Original: Train the Q t on N computation nodes and Evaluate the accuracy of Q t . P t +1 ← Q t ∪ P t 10: t = t +11: end while Output The Pareto Optimal Front A ∗ i another; 2) change the output channel of one layer; 3) Add/remove/modify a skip connection. We keep updating the Pareto Front between step time and accuracy during the whole process.
Revised: 505.00 -21111111211111ET-NAS-L 23311-a02c12 64 211130.

Original: As a result, 10 blocks are selected as the candidates for the following rounds.
Revised: 542.52 -21111111111111111111111result, 10 blocks are selected as the candidates for the following rounds.

Original: Practically, during our search, the performance of early stop models aligns well with the fully-train accuracy.
Revised: Practically, during our search, the performance of early stop models aligns well with the fully-train accuracy.

Original: We checked the Spearman Rank Correlation for 103 architectures: ρ = 96 .
Revised: We checked the Spearman Rank Correlation for 103 architectures: ρ = 96 .

Original: Thus, using early stop can greatly reduce the search cost by around 90% while keeping our NAS effective.[Figure]
Revised: Thus, using early stop can greatly reduce the search cost by around 90% while keeping our NAS effective.

Original:  M 
Revised: B.3. M ACRO - LEVEL SEARCH

Original: [Figure] searchthe block-level architectures withlevel search
Revised: search with

Original: Random searchat ﬁrst, where the number of blocks is chosen randomly between10 and50, and the ﬁrst
Revised: -level search is chosen ran- between and

Original: and last channel is drawn from { 32 , 64 , 128 } and { 512 , 1024 , 2048 } respectively.
Revised: } respectively.

Original: EA search is then applied, where the mutations allowed are: 1) Add a ‘1’; 2) Remove a ‘1’; 3) Swap two different adjacent numbers.
Revised: EA search is then applied, where the mutations allowed are: 1) Add a ‘1’; 2) Remove a ‘1’; 3) Swap two different adjacent numbers.

Original: Similar to block-level search, Pareto Front between step time and accuracy is also kept updated.
Revised: Similar to block-level search, Pareto Front between step time and accuracy is also kept updated.

Original: Different from the previous phase, the whole ImageNet dataset is utilized for training.
Revised: Different from the previous phase, the whole ImageNet dataset is utilized for training.

Original: Each model is trained with a batch size of 1024 and a learning rate of 0.2 for 40 epochs.
Revised: Each model is trained with a batch size of 1024 and a learning rate of 0.2 for 40 epochs.

Original: B. ET-NAS: M ODEL Z OO I NFORMATION AND THEIR E NCODINGS
Revised: B. ET-NAS: M ODEL Z OO I NFORMATION AND THEIR E NCODINGS

Original: After the NAS process done in Section
Revised: After the NAS process done in Section

Original: B.3, 12 models are selected as our model zoo ET-NAS of ﬁne-tuning.
Revised: B.3, 12 models are selected as our model zoo ET-NAS of ﬁne-tuning.

Original: Details of these models are shown in Table B.4.
Revised: Details of these models are shown in Table B.4.

Original: The inference time and step time are measured in ms on a single Nvidia V100, with batch size of 64.
Revised: The inference time and step time are measured in ms on a single Nvidia V100, with batch size of 64.

Original: The resolution follows the standard setting of ImageNet: 224x224.
Revised: The resolution follows the standard setting of ImageNet: 224x224.

Original: By observing the optimal model in the table, smaller models should use simpler blocks while bigger models prefer complex blocks. 
Revised: By observing the optimal model in the table, smaller models should use simpler blocks while bigger models prefer complex blocks. [Figure] [Figure]

Revised: n=5500 R-sq=56%

Revised: Macro-levelRegressionAnalysis n=1200 R-sq=71%

Revised: Terms

Original: Figure 9 shows the comparison of our ET-NAS models with other SOTA ImageNet models.
Revised: Coef SECoef T-Value P-Value Terms Coef SECoef T-Value P-Value OP1ChannelChangeRatio -0.183 0.

Revised: ChannelSize -0.210 0.

Revised: OP2ChannelChangeRatio -0.168 0.

Revised: DoubleChannelPosition1 -0.110 0.

Revised: Numofskipconnection(add) -0.272 0.

Revised: DoubleChannelPosition2 0.

Revised: Numofskipconnection(concat) -0.362 0.

Revised: DoubleChannelPosition3 -0.016 0.

Revised: Output channel -0.539 0.

Revised: DoubleChannelPosition4 0.

Revised: conv3x3(ref) 0.

Revised: DoubleChannelPosition5 0.

Revised: conv1x -0.037 0.

Revised: NumblockinStage- -0.562 0.

Revised: conv3x3,wgroup= 0.

Revised: NumblockinStage- -0.139 0.

Revised: conv3x3,wgroup= 0.

Revised: NumblockinStage0.

Revised: Separableconv3x -0.

Revised: [Figure]

Revised: Figure 10 shows the comparison of our ET-NAS models with other SOTA ImageNet models.

Original: Inference time and training step time are measured in ms on a single Nvidia V100, with bs = 64 .
Revised: Inference time and training step time are measured in ms on a single Nvidia V100, with bs = 64 .

Original: Our
Revised: Our

Original: ET-NAS series show superior performance comparing to RegNet, EfﬁcientNet series.
Revised: ET-NAS series show superior performance comparing to RegNet, EfﬁcientNet series.

Original: Comparing to some EA-based NAS methods such as OFANet and Amoebanet, our method is also efﬁcient in [Table] [Figure] [Table]
Revised: Comparing to some EA-based NAS methods such as OFANet and Amoebanet, our method is also efﬁcient in terms of training.

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table] [Figure] terms of training. We found that there exists a performance ranking gap between inference time and training step time in Figure 9.
Revised: We found that there exists a performance ranking gap between inference time and training step time in Figure 10.

Original: This is mainly due to the depth and the main type of operation of the models.
Revised: This is mainly due to the depth and the main type of operation of the models.

Original: We found that deeper networks with separable conv such as EfﬁcientNet/MobileNet have a larger training-step-time/inference-time ratio comparing to our models (shallower&more common conv).
Revised: We found that deeper networks with separable conv such as EfﬁcientNet/MobileNet have a larger training-step-time/inference-time ratio comparing to our models (shallower&more common conv).

Original: B.4. W HAT MAKES A NETWORK EFFICIENT - TRAINING ?
Revised: B.4. W HAT MAKES A NETWORK EFFICIENT - TRAINING ?

Original: To answer this question, we ﬁrst need to deﬁne a score for the efﬁciency of the searched models A .
Revised: To answer this question, we ﬁrst need to deﬁne a score for the efﬁciency of the searched models A .

Original: In
Revised: In

Original: MOOP, the goodness of a solution is determined by dominance .
Revised: MOOP, the goodness of a solution is determined by dominance .

Original: Thus, we can use the non-dominated sorting algorithm to sort the A according to the Pareto dominance principle.
Revised: Thus, we can use the non-dominated sorting algorithm to sort the A according to the Pareto dominance principle.

Original: Each architecture is assigned to one Pareto front and the rank R P of that Pareto front can be regarded as the goodness of a solution, in our case, the efﬁciency.
Revised: Each architecture is assigned to one Pareto front and the rank R P of that Pareto front can be regarded as the goodness of a solution, in our case, the efﬁciency.

Original: We then deﬁned the efﬁciency score of A as: s E ( A ) =−
Revised: We then deﬁned the efﬁciency score of A as: s E ( A ) =

Original: [Equation]
Revised: [Equation]

Original: Since Pareto optimal front is the Rank 1 Pareto front , larger efﬁciency score s E ( A ) means better efﬁciency.[Table]
Revised: Since Pareto optimal front is the Rank 1 Pareto front , larger efﬁciency score s E ( A ) means better efﬁciency.

Original: NumblockinStage- -0.139 0.

Original: conv3x3,wgroup= 0.

Original: NumblockinStage0.

Original: Separableconv3x -0.200 0.

Original: NumblockinStage- -0.010 0.

Original: Then we perform a multivariate linear regression analysis on the A S .
Revised: Then we perform a multivariate linear regression analysis on the A S .

Original: According to our search space, ordinal/nominal variables that describe the model are denoted as predictors to ﬁt the s E ( A ) .
Revised: According to our search space, ordinal/nominal variables that describe the model are denoted as predictors to ﬁt the s E ( A ) .

Original: Table 11 shows the coefﬁcients from the regression analysis on both block-level and macro-level designs.
Revised: Table 11 shows the coefﬁcients from the regression analysis on both block-level and macro-level designs.

Original: Positive coefﬁcients indicate a positive relationship.
Revised: Positive coefﬁcients indicate a positive relationship.

Original: “P-Value” shows the signiﬁcance of the variables.
Revised: “P-Value” shows the signiﬁcance of the variables.

Original: We summarize and highlight several noteworthy conclusions uncovered by our analysis:
Revised: We summarize and highlight several noteworthy conclusions uncovered by our analysis:

Original: • By observing optimal A ∗ i , smaller models should use simpler blocks while bigger models prefer complex blocks.
Revised: • By observing optimal A ∗ i , smaller models should use simpler blocks while bigger models prefer complex blocks.

Original: Simply increasing depth/width to expand the model in Tan & Le (2019) may not be optimal.
Revised: Simply increasing depth/width to expand the model in Tan & Le (2019) may not be optimal.

Original: • Adding additional skip connections will decrease the training efﬁciency of the model (The Coef is signiﬁcantly negative).
Revised: • Adding additional skip connections will decrease the training efﬁciency of the model (The Coef is signiﬁcantly negative).

Original: Using “add” to combine the features is more efﬁcient than “concat”.
Revised: Using “add” to combine the features is more efﬁcient than “concat”.

Original: • Using “conv3x3, w group=4” is the best operation among the searched operations (Coef is 0.295).
Revised: • Using “conv3x3, w group=4” is the best operation among the searched operations (Coef is 0.295).

Original: Separable conv3x3 is not efﬁcient for training (Coef is -0.2).
Revised: Separable conv3x3 is not efﬁcient for training (Coef is -0.2).

Original: • The ﬁrst double-channel position should be more close to the beginning of the network, while the ﬁnal double channel-position should be delayed to the end of the network.
Revised: • The ﬁrst double-channel position should be more close to the beginning of the network, while the ﬁnal double channel-position should be delayed to the end of the network.

Original: • Fewer blocks should be assigned to the ﬁrst two stages.
Revised: • Fewer blocks should be assigned to the ﬁrst two stages.

Original: More should be assigned to the 3rd stage.
Revised: More should be assigned to the 3rd stage.

Original: B. CO 2 CONSUMPTION A NALYSIS
Revised: B. CO 2 CONSUMPTION A NALYSIS

Original: Fine-tuning from the pretrained ImageNet/language model is a de-facto practice in the deep learning ﬁeld (CV/NLP).
Revised: Fine-tuning from the pretrained ImageNet/language model is a de-facto practice in the deep learning ﬁeld (CV/NLP).

Original: Our NASOA improves the efﬁciency of ﬁne-tuning which has the potentials to greatly reduce computational cost in GPU clusters/cloud computing.
Revised: Our NASOA improves the efﬁciency of ﬁne-tuning which has the potentials to greatly reduce computational cost in GPU clusters/cloud computing.

Original: According to a recent study (Strubell et al., 2019b), developing and tuning for one typical R&D project (Strubell et al., 2018) in
Revised: According to a recent study (Strubell et al., 2019b), developing and tuning for one typical R&D project (Strubell et al., 2018) in

Original: Google Cloud computing needs about $250k cost, 82k kWh electricity, and 123k lbs CO 2 emission, which equals to the CO 2 consumption of air traveling (NY ↔ SF) 62 times.
Revised: Google Cloud computing needs about $250k cost, 82k kWh electricity, and 123k lbs CO 2 emission, which equals to the CO 2 consumption of air traveling (NY ↔ SF) 62 times.

Original: Among most of them, 123 hyperparameter grid searches were performed for new datasets, resulting in 4789 jobs in total.
Revised: Among most of them, 123 hyperparameter grid searches were performed for new datasets, resulting in 4789 jobs in total.

Original: It is believed that the proposed faster ﬁne-tuning pipeline can save up to 40x computational cost among them.
Revised: It is believed that the proposed faster ﬁne-tuning pipeline can save up to 40x computational cost among them.

Original: Furthermore, we have released all the searched efﬁcient models to help the public skipping the computation-heavy NAS stage and directly enjoy the beneﬁt of our methods.
Revised: Furthermore, we have released all the searched efﬁcient models to help the public skipping the computation-heavy NAS stage and directly enjoy the beneﬁt of our methods.

Original: In conclusion, our
Revised: In conclusion, our

Original: NASOA is meaningful for environment protection and energy saving.
Revised: NASOA is meaningful for environment protection and energy saving.

Original: B. D ETAILED A LGORITHMS OF NASOA
Revised: B. D ETAILED A LGORITHMS OF NASOA

Original: Detailed algorithms of our Model Zoo (ET-NAS) search can be found in Algorithm 2.
Revised: Detailed algorithms of our Model Zoo (ET-NAS) search can be found in Algorithm 2.

Original: The pseudo code of online ﬁne-tuning schedule generator training, prediction, and update can be found in Algorithm 3.
Revised: The pseudo code of online ﬁne-tuning schedule generator training, prediction, and update can be found in Algorithm 3.

Original: We use BOHB and random search in our experiments as the HPO baseline. As stated in Section 4.4, The shortest/longest time constraint (budget) is deﬁned as the time of ﬁne-tuning 10/50 epochs for
Revised: Algorithm

Original: Input: Block/Macro Search Space S
Revised: Input: Block/Macro Search Space S

Original: i , S a , Stop Criterion Γ , # Computation Nodes K , Sensitive
Revised: i , S a , Stop Criterion Γ , # Computation Nodes K , Sensitive

Original: Factor (cid:15) , # Block Architectures M , # Models in Model Zoo N .
Revised: Factor (cid:15) , # Block Architectures M , # Models in Model Zoo N .

Original: Output: Final Model Zoo Z oo 1: procedure B LOCK S EARCH ( S i , Γ , K, (cid:15), M ) 2:
Revised: Output: Final Model Zoo Z oo 1: procedure B LOCK S EARCH ( S i , Γ , K, (cid:15), M ) 2:

Original: P f ← NSGA-II( Γ , S i , K, (cid:15) )
Revised: P f ← NSGA-II( Γ , S i , K, (cid:15) )

Original: (cid:46)
Revised: (cid:46)

Original: Our modiﬁed NSGA-II, see Algorithm3:
Revised: Our modiﬁed NSGA-II, see Algorithm3:

Original: Cells ← M OST C OMMON ( P f , M ) (cid:46)
Revised: Cells ← M OST C OMMON ( P f , M ) (cid:46)

Original: Most common M cells from P f 4: end procedure 5: procedure M ACRO S EARCH ( S a , Γ , K, (cid:15), N ) 6:
Revised: Most common M cells from P f 4: end procedure 5: procedure M ACRO S EARCH ( S a , Γ , K, (cid:15), N ) 6:

Original: P f ← NSGA-II( Γ , S a ( Cells ) , K, (cid:15) ) 7:
Revised: P f ← NSGA-II( Γ , S a ( Cells ) , K, (cid:15) ) 7:

Original: Z oo ← N SGA S ORT ( P f , N, (cid:15) )
Revised: Z oo ← N SGA S ORT ( P f , N, (cid:15) )

Original: (cid:46)
Revised: (cid:46)

Original: Choose models based on crowding-distance 8: end procedure
Revised: Choose models based on crowding-distance 8: end procedure

Original: Input: Model Zoo Z oo , Time Evaluator T s , Acc Evaluator T r , Hyper-parameter Space
Revised: We use BOHB and random search in our experiments as the HPO baseline. As stated in Section 4.4,

Original: S HP , Known Datasets D old , New Dataset D new , # Meta-data H M , Time Constraint T l , # Conﬁgurations H .
Revised: The shortest/longest time constraint (budget) is deﬁned as the time of ﬁne-tuning 10/50 epochs for ResNet18/ResNet101 and the rest are equally divided into the log-space, which can be represented as: t x = t 0 ∗ ( t 3 /t 0 ) x/t 3 , where x = [0 , 1 , 2 , 3] , t 0 is the time to train ResNet18 for 10 epochs and t 3 is the time to train ResNet101 for 50 epochs. We only compare the HPO setting under the same max computational budgets equal to t 1 in Table 5 (left).

Original: Output: Optimal Model A ∗ , Hyper-parameters Regime ∗ FT , Predictor Acc P 1: procedure O FFLINE T RAINING (Z oo , T r , S HP , D old , H M ) 2:
Revised: For random search, we randomly sample candidates from predeﬁned search space until Input: Output 1: procedure O FFLINE T RAINING (

Original:  MetaData ← ∅ , Acc P ← A DAPTIVE MLP(.)
Revised: S D MetaData ← ∅ , Acc P ← A DAPTIVE MLP(.)

Original: (cid:46) Initialize default predictor 3: for D ∈ D old , i ← 1 to H M do 4:
Revised: (cid:46) Initialize default predictor 3: for D ∈ D old , i ← 1 to H M do 4:

Original: A , Regime FT ← R ANDOM ( Z oo , S HP ) (cid:46) Randomly select from search space 5:
Revised: A , Regime FT ← R ANDOM ( Z oo , S HP ) (cid:46) Randomly select from search space 5:

Original: Acc ← T r ( D, A , Regime FT ) (cid:46) Train with selected conﬁguration 6:
Revised: Acc ← T r ( D, A , Regime FT ) (cid:46) Train with selected conﬁguration 6:

Original: MetaData ← MetaData ∪ { ( A , Regime FT , Acc ) } (cid:46) Add this result to meta-data 7: end for 8:
Revised: MetaData ← MetaData ∪ { ( A , Regime FT , Acc ) } (cid:46) Add this result to meta-data 7: end for 8:

Original: Acc P ← T RAIN ( Acc P , MetaData ) (cid:46) Train predictor with all meta-data 9: end procedure 10: procedure O NLINE P REDICTION ( Z oo , D new , Acc P , T l , T s , H ) 11:
Revised: Acc P ← T RAIN ( Acc P , MetaData ) (cid:46) Train predictor with all meta-data 9: end procedure 10: procedure O NLINE P REDICTION ( Z oo , D new , Acc P , T l , T s , H ) 11:

Original: MetaData ← ∅ 12: for i ← 1 to H do 13:
Revised: MetaData ← ∅ 12: for i ← 1 to H do 13:

Original: A ← R ANDOM ( Z oo ) 14:
Revised: A ← R ANDOM ( Z oo ) 14:

Original: Epoch ← T l
Revised: Epoch ← T l

Original: ÷ T s ( A , D new ) (cid:46)
Revised: ÷ T s ( A , D new ) (cid:46)

Original: Always choose the largest epoch within T l 15:
Revised: Always choose the largest epoch within T l 15:

Original: Regime FT ← R ANDOM ( S HP | Epoch )
Revised: Regime FT ← R ANDOM ( S HP | Epoch )

Original: (cid:46) Randomly select condition on Epoch 16:
Revised: (cid:46) Randomly select condition on Epoch 16:

Original: MetaData ← MetaData ∪ { ( A , Regime FT ) } 17: end for 18:
Revised: MetaData ← MetaData ∪ { ( A , Regime FT ) } 17: end for 18:

Original: A ∗ , Regime ∗ FT ← P REDICT ( G, MetaData )
Revised: A ∗ , Regime ∗ FT ← P REDICT ( G, MetaData )

Original: (cid:46) Choose the optimal from H conﬁgs 19:
Revised: (cid:46) Choose the optimal from H conﬁgs 19:

Original: Acc ← T r ( D new , A ∗ , Regime ∗ FT ) 20: 
Revised: Acc ← T r ( D new , A ∗ , Regime ∗ FT ) 20: [Table] [Table]

Original: Acc P ← T RAIN ( Acc P , {A ∗ , Regime ∗ FT , Acc } ) (cid:46) Improve predictor with this meta-data 21: end procedure
Revised: [Table] [Table] , Acc } ) (cid:46) Improve predictor with this meta

Original: ResNet18/ResNet101 and the rest are equally divided into the log-space, which can be represented as: t x = t 0 ∗ ( t 3 /t 0 ) x/t 3 , where x = [0 , 1 , 2 , 3] , t 0 is the time to train ResNet18 for 10 epochs and t 3 is the time to train ResNet101 for 50 epochs.
Revised: [Table]

Original: We only compare the HPO setting under the same max computational budgets equal to t 1 in Table 5 (left).
Revised: data [Table]

Original: For random search, we randomly sample candidates from predeﬁned search space until reaching the max computational budget.
Revised: procedure reaching the max computational budget.

Original: And for BOHB, we use the opensource implementation of BOHB at https://github.com/automl/HpBandSter .
Revised: And for BOHB, we use the opensource implementation of BOHB at https://github.com/automl/HpBandSter .

Original: We set random fraction=0.3, percent of good observations=15%, min budget=25% and max budget=100% with respect to our max computational budget.
Revised: We set random fraction=0.3, percent of good observations=15%, min budget=25% and max budget=100% with respect to our max computational budget.

Original: And we use BOHB with 40x computational cost than our proposed methods with different model zoos.
Revised: And we use BOHB with 40x computational cost than our proposed methods with different model zoos.

Original: The results are presented in Section 4.4.
Revised: The results are presented in Section 4.4.