Original: Efficient Architecture Search for Diverse Tasks
Revised: Efﬁcient Architecture Search for Diverse Tasks

Original: Abstract
Revised: Abstract

Original: While neural architecture search (NAS) has enabled automated machine learning (AutoML) for well-researched areas, its application to tasks beyond computer vision is still under-explored.
Revised: While neural architecture search (NAS) has enabled automated machine learning (AutoML) for well-researched areas, its application to tasks beyond computer vision is still under-explored.

Original: As less-studied domains are precisely those where we expect
Revised: As less-studied domains are precisely those where we expect

Original: AutoML to have the greatest impact, in this work we study NAS for efficiently solving diverse problems.
Revised: AutoML to have the greatest impact, in this work we study NAS for efﬁciently solving diverse problems.

Original: Seeking an approach that is fast, simple, and broadly applicable, we fix a standard convolutional network (CNN) topology and propose to search for the right kernel sizes and dilations its operations should take on.
Revised: Seeking an approach that is fast, simple, and broadly applicable, we ﬁx a standard convolutional network (CNN) topology and propose to search for the right kernel sizes and dilations its operations should take on.

Original: This dramatically expands the model’s capacity to extract features at multiple resolutions for different types of data while only requiring search over the operation space.
Revised: This dramatically expands the model’s capacity to extract features at multiple resolutions for different types of data while only requiring search over the operation space.

Original: To overcome the efficiency challenges of naive weight-sharing in this search space, we introduce D ASH , a differentiable NAS algorithm that computes the mixtureof-operations using the Fourier diagonalization of convolution, achieving both a better asymptotic complexity and an up-to-10x search time speedup in practice.
Revised: To overcome the efﬁciency challenges of naive weight-sharing in this search space, we introduce D ASH , a differentiable NAS algorithm that computes the mixtureof-operations using the Fourier diagonalization of convolution, achieving both a better asymptotic complexity and an up-to-10x search time speedup in practice.

Original: We evaluate D ASH on ten tasks spanning a variety of application domains such as PDE solving, protein folding, and heart disease detection.
Revised: We evaluate D ASH on ten tasks spanning a variety of application domains such as PDE solving, protein folding, and heart disease detection.

Original: D ASH outperforms state-of-the-art AutoML methods in aggregate, attaining the best-known automated performance on seven tasks.
Revised: D ASH outperforms state-of-the-art AutoML methods in aggregate, attaining the best-known automated performance on seven tasks.

Original: Meanwhile, on six of the ten tasks, the combined search and retraining time is less than 2x slower than simply training a CNN backbone that is far less accurate.
Revised: Meanwhile, on six of the ten tasks, the combined search and retraining time is less than 2x slower than simply training a CNN backbone that is far less accurate.

Original:  Introduction
Revised:  Introduction

Original: The success of deep learning for computer vision and natural language processing has spurred growing interest in enabling similar breakthroughs for other domains such as biology, healthcare, and physical sciences.
Revised: The success of deep learning for computer vision and natural language processing has spurred growing interest in enabling similar breakthroughs for other domains such as biology, healthcare, and physical sciences.

Original: Consequently, there is enormous potential for neural architecture search to help automate model development in these diverse areas.
Revised: Consequently, there is enormous potential for neural architecture search to help automate model development in these diverse areas.

Original: However, while extensive NAS research has been devoted to improving the search speed [1, 2] and automatically attaining state-of-the-art performance on vision datasets such as CIFAR and ImageNet [3], the resulting algorithms have subpar performance beyond the tasks on which they were developed.
Revised: However, while extensive NAS research has been devoted to improving the search speed [1, 2] and automatically attaining state-of-the-art performance on vision datasets such as CIFAR and ImageNet [3], the resulting algorithms have subpar performance beyond the tasks on which they were developed.

Original: For example, in the analysis of
Revised: For example, in the analysis of

Original: NAS-Bench-360 [4], a recent benchmark designed for improving task diversity in NAS evaluation, the authors showed a significant gap between models found by NAS methods, such as DARTS [2] and DenseNAS [5], and hand-crafted expert architectures on a number of distinct applications.
Revised: NAS-Bench-360 [4], a recent benchmark designed for improving task diversity in NAS evaluation, the authors showed a signiﬁcant gap between models found by NAS methods, such as DARTS [2] and DenseNAS [5], and hand-crafted expert architectures on a number of distinct applications.

Original: To improve the generalizability of AutoML methods, recent works such as AutoML-Zero [6] and XDoperations [7] propose to relax the inductive biases encoded in the standard search spaces.
Revised: To improve the generalizability of AutoML methods, recent works such as AutoML-Zero [6] and XDoperations [7] propose to relax the inductive biases encoded in the standard search spaces.

Original: However, the former is not designed for practical deployment, and the latter is too expensive to execute even for simple problems like CIFAR-100 (Fig. 1b).
Revised: However, the former is not designed for practical deployment, and the latter is too expensive to execute even for simple problems like CIFAR-100 (Fig. 1b).

Original: Hence we ask: is there an approach that can provide sufficient expressivity to yield high accuracy across multiple domains, while still retaining the faster search and the more efficient final models of discrete architecture search?
Revised: Hence we ask: is there an approach that can provide sufﬁcient expressivity to yield high accuracy across multiple domains, while still retaining the faster search and the more efﬁcient ﬁnal models of discrete architecture search?

Original: ∗ Equal contribution.[Figure]
Revised: ⇤ Equal contribution.

Original: F r a c t i o n o f - S u b o p t i m a l T a s k s Best [Figure]
Revised: 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Original: Hand

Original: [Figure]
Revised: [Figure]

Original: DASH (a)

Original: WRN DARTS XD DASH

Original: C

Original: I F

Original: A R100 R un t i m e (

Original: G P U H o u r s ) (b) In this work, we answer in the affirmative by introducing a novel NAS method called
Revised: In this work, we answer in the afﬁrmative by introducing a novel NAS method called

Original: D ASH ( D iverse-task
Revised: D ASH ( D iverse-task

Original: A rchitecture S earc H ).
Revised: A rchitecture S earc H ).

Original: In order to attain multi-domain capability , D ASH adapts standard CNN backbones to various learning problems by finding substitutes for their layer operations.
Revised: In order to attain multi-domain capability , D ASH adapts standard CNN backbones to various learning problems by ﬁnding substitutes for their layer operations.

Original: Concretely, we consider a search space of cross-scale dilated convolutions which are effective for multi-scale feature extraction [8, 9] and context aggregation [10, 11].
Revised: Concretely, we consider a search space of cross-scale dilated convolutions which are effective for multi-scale feature extraction [8, 9] and context aggregation [10, 11].

Original: Our key difference from past search spaces is that we explicitly consider filters with a wide range of kernel sizes and dilations —while most NAS methods only handle kernels with maximum size 5 and dilation rate 2 , our proposed operator space includes not only the conventional small kernels but also significantly larger ones with size 15 or dilation 127 (Fig. 2).
Revised: Our key difference from past search spaces is that we explicitly consider ﬁlters with a wide range of kernel sizes and dilations —while most NAS methods only handle kernels with maximum size 5 and dilation rate 2 , our proposed operator space includes not only the conventional small kernels but also signiﬁcantly larger ones with size 15 or dilation 127 (Fig. 2).

Original: This design choice is motivated by the fact that large kernels can capture input relations for dense prediction problems [12], model long-range dependencies for sequence tasks [13, 14], and resemble global-attention in Transformers [15].
Revised: This design choice is motivated by the fact that large kernels can capture input relations for dense prediction problems [12], model long-range dependencies for sequence tasks [13, 14], and resemble global-attention in Transformers [15].

Original: Thus, D ASH ’s cross-scale search space enables adaptation to diverse downstream tasks, unlike prior NAS work which targets image classification and assumes that small kernels are sufficient.
Revised: Thus, D ASH ’s cross-scale search space enables adaptation to diverse downstream tasks, unlike prior NAS work which targets image classiﬁcation and assumes that small kernels are sufﬁcient.

Original: However, efficiently searching for an appropriate kernel configuration in this expansive cross-scale search space is non-trivial.
Revised: However, efﬁciently searching for an appropriate kernel conﬁguration in this expansive cross-scale search space is non-trivial.

Original: Indeed, for existing NAS algorithms, the cost of exploring a combinatorially large set of operators is substantial.
Revised: Indeed, for existing NAS algorithms, the cost of exploring a combinatorially large set of operators is substantial.

Original: Even for weight-sharing methods that are known for efficiency, e.g., DARTS [16], the computational complexity scales directly with the number of kernels considered and quadratically with the largest kernel size.
Revised: Even for weight-sharing methods that are known for efﬁciency, e.g., DARTS [16], the computational complexity scales directly with the number of kernels considered and quadratically with the largest kernel size.

Original: To overcome this obstacle, D ASH explores multi-scale convolutions via three techniques—the first two exploit mathematical properties of convolutions, and the last one takes advantage of fast matrix multiplication on GPUs.
Revised: To overcome this obstacle, D ASH explores multi-scale convolutions via three techniques—the ﬁrst two exploit mathematical properties of convolutions, and the last one takes advantage of fast matrix multiplication on GPUs.

Original: Specifically:
Revised: Speciﬁcally:

Original: Using the linearity of convolutions, we mix several convolutions by computing one convolution equipped with a combined kernel rather than applying each filter separately and aggregating multiple outputs.
Revised: Using the linearity of convolutions, we mix several convolutions by computing one convolution equipped with a combined kernel rather than applying each ﬁlter separately and aggregating multiple outputs.

Original: While the number of convolution computations required by the naive aggregation of | K | possible kernel sizes and | D | possible dilations is O ( | K || D | ) , our approach has O (1) complexity, independent of the search space size.
Revised: While the number of convolution computations required by the naive aggregation of | K | possible kernel sizes and | D | possible dilations is O ( | K || D | ) , our approach has O (1) complexity, independent of the search space size.

Original: Using the diagonalization of convolutions, we relegate a major portion of the computation to element-wise multiplication in the Fourier domain, minimizing the effect of the largest kernel size on the complexity of our algorithm.
Revised: Using the diagonalization of convolutions, we relegate a major portion of the computation to element-wise multiplication in the Fourier domain, minimizing the effect of the largest kernel size on the complexity of our algorithm.

Original: For instance, a standard 1D convolution requires O ( nk ) operations to convolve a size- k kernel with a length- n input, but a Fourier convolution takes only O ( n log n ) , a critical improvement that makes searching over large kernels significantly easier.
Revised: For instance, a standard 1D convolution requires O ( nk ) operations to convolve a size- k kernel with a length- n input, but a Fourier convolution takes only O ( n log n ) , a critical improvement that makes searching over large kernels signiﬁcantly easier.

Original: Our final strategy is to use Kronecker products of undilated kernels and small sparse matrices to compute dilated kernels quickly on GPUs.
Revised: Our ﬁnal strategy is to use Kronecker products of undilated kernels and small sparse matrices to compute dilated kernels quickly on GPUs.

Original: This brings an additional two-fold speedup on top of the previous techniques.
Revised: This brings an additional two-fold speedup on top of the previous techniques.

Original: Aside from these innovations, D ASH employs the standard weight-sharing scheme of training a supernet, discretizing to obtain a model, and retraining the model for end tasks [2].
Revised: Aside from these innovations, D ASH employs the standard weight-sharing scheme of training a supernet, discretizing to obtain a model, and retraining the model for end tasks [2].

Original: We analyze the asymptotic complexity of the first two techniques and verify the practical utility when all three are combined together.
Revised: We analyze the asymptotic complexity of the ﬁrst two techniques and verify the practical utility when all three are combined together.

Original: In particular, D ASH achieves a ten-fold speedup in total for differentiable NAS over the multi-scale search space.
Revised: In particular, D ASH achieves a ten-fold speedup in total for differentiable NAS over the multi-scale search space.

Original: Moreover, we show that searching over large kernels is necessary to solve diverse problems and that each technique on its own cannot scale in this large-kernel setting.
Revised: Moreover, we show that searching over large kernels is necessary to solve diverse problems and that each technique on its own cannot scale in this large-kernel setting.

Original: In terms of accuracy performance, we evaluate D ASH on ten datasets spanning multiple application areas such as PDE solving, protein folding, and disease prediction from NAS-Bench-360 [4].
Revised: In terms of accuracy performance, we evaluate D ASH on ten datasets spanning multiple application areas such as PDE solving, protein folding, and disease prediction from NAS-Bench-360 [4].

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, D ASH yields models with better aggregate performance than those returned by leading AutoML methods as well as hand-designed task-specific architectures.
Revised: 1, D ASH yields models with better aggregate performance than those returned by leading AutoML methods as well as hand-designed task-speciﬁc architectures.

Original: As for individual tasks,
Revised: As for individual tasks,

Original: D ASH beats all past automated approaches on seven of the ten problems and exceeds hand-designed models on seven, simultaneously maintaining strong efficiency relative to weight-sharing methods like DARTS.
Revised: D ASH beats all past automated approaches on seven of the ten problems and exceeds hand-designed models on seven, simultaneously maintaining strong efﬁciency relative to weight-sharing methods like DARTS.

Original: The empirical success of D ASH implies that CNNs with appropriate kernels can be competitive baselines for problems where expert architectures are not available.
Revised: The empirical success of D ASH implies that CNNs with appropriate kernels can be competitive baselines for problems where expert architectures are not available.

Original: Our code is made public at https://github.com/sjunhongshen/DASH .
Revised: Our code is made public at https://github.com/sjunhongshen/DASH .

Original:  Related Work
Revised:  Related Work

Original: Neural architecture search (NAS) aims to automate the design of neural networks.
Revised: Neural architecture search (NAS) aims to automate the design of neural networks.

Original: Recently, there has been significant progress in both search space design [17, 2, 7] and search strategy development [1, 18, 2, 19, 3].
Revised: Recently, there has been signiﬁcant progress in both search space design [17, 2, 7] and search strategy development [1, 18, 2, 19, 3].

Original: However, these methods are mostly evaluated on image classification or segmentation, with a few focusing on new applications such as image restoration [20], audio classification [21], and machine translation [22].
Revised: However, these methods are mostly evaluated on image classiﬁcation or segmentation, with a few focusing on new applications such as image restoration [20], audio classiﬁcation [21], and machine translation [22].

Original: Problems beyond the vision and language domains are less-explored.
Revised: Problems beyond the vision and language domains are less-explored.

Original: In this work, we seek to improve the generalizability of NAS by a morphism-based approach [23, 24, 7] that adapts existing CNN backbones to target tasks.
Revised: In this work, we seek to improve the generalizability of NAS by a morphism-based approach [23, 24, 7] that adapts existing CNN backbones to target tasks.

Original: We specifically focus on expanding the operation space to multiple types of convolutions, simultaneously varying the kernel size and the dilation factor.
Revised: We speciﬁcally focus on expanding the operation space to multiple types of convolutions, simultaneously varying the kernel size and the dilation factor.

Original: Past work in related directions has at most studied the easier problem of altering dilation alone, and only for vision tasks [25].
Revised: Past work in related directions has at most studied the easier problem of altering dilation alone, and only for vision tasks [25].

Original: Therefore, although convolution has been an integral part of NAS, how to search over a large set of convolutional operators remains an open problem.
Revised: Therefore, although convolution has been an integral part of NAS, how to search over a large set of convolutional operators remains an open problem.

Original: In the following, we identify four types of solutions from existing work and illustrate their limitations.
Revised: In the following, we identify four types of solutions from existing work and illustrate their limitations.

Original: Differentiable Architecture Search (DARTS)
Revised: Differentiable Architecture Search (DARTS)

Original: We can treat convolutions as ordinary operators and apply a scalable NAS algorithm.
Revised: We can treat convolutions as ordinary operators and apply a scalable NAS algorithm.

Original: In particular, DARTS [2] introduces continuous relaxation to the weight-sharing paradigm [1] and allows us to gain information about many networks efficiently by training a combined supernet.
Revised: In particular, DARTS [2] introduces continuous relaxation to the weight-sharing paradigm [1] and allows us to gain information about many networks efﬁciently by training a combined supernet.

Original: The algorithm relaxes the discrete set of operations at each edge in a computational graph as a softmax so the search process is end-to-end differentiable and amenable to regular optimizers.
Revised: The algorithm relaxes the discrete set of operations at each edge in a computational graph as a softmax so the search process is end-to-end differentiable and amenable to regular optimizers.

Original: After search, it discretizes the weights to output a valid architecture.
Revised: After search, it discretizes the weights to output a valid architecture.

Original: The original
Revised: The original

Original: DARTS search space contains only four convolutions with kernel sizes no larger than 5 and dilation rates no larger than 2 .
Revised: DARTS search space contains only four convolutions with kernel sizes no larger than 5 and dilation rates no larger than 2 .

Original: While this small search space might be enough for low-resolution image input, it is insufficient for diverse tasks such as high-dimensional time series problems [4].
Revised: While this small search space might be enough for low-resolution image input, it is insufﬁcient for diverse tasks such as high-dimensional time series problems [4].

Original: Although one could add more convolutions one-by-one to the operator set to augment performance, this approach scales poorly, as reflected in the limited search spaces of similar methods like AMBER [26].
Revised: Although one could add more convolutions one-by-one to the operator set to augment performance, this approach scales poorly, as reﬂected in the limited search spaces of similar methods like AMBER [26].

Original: In fact,
Revised: In fact,

Original: AMBER has to shift the kernel size up to achieve good performance on long-sequence genomic data.
Revised: AMBER has to shift the kernel size up to achieve good performance on long-sequence genomic data.

Original: MergeNAS and RepVGG An alternative way to explore the convolutional search space is to take advantage of the operator’s linearity.
Revised: MergeNAS and RepVGG An alternative way to explore the convolutional search space is to take advantage of the operator’s linearity.

Original: That is, we can first mix the kernels and then apply convolution once, unlike DARTS which computes each convolution separately and outputs the aggregated result to the next layer.
Revised: That is, we can ﬁrst mix the kernels and then apply convolution once, unlike DARTS which computes each convolution separately and outputs the aggregated result to the next layer.

Original: This kernel-mixing strategy, which we call mixed-weights and will formally define in Section 3.2, has been employed by MergeNAS [27] and RepVGG [28] to improve architecture search robustness and VGG inference speed, respectively.
Revised: This kernel-mixing strategy, which we call mixed-weights and will formally deﬁne in Section 3.2, has been employed by MergeNAS [27] and RepVGG [28] to improve architecture search robustness and VGG inference speed, respectively.

Original: It works well for a few small kernels.
Revised: It works well for a few small kernels.

Original: However, we will show later that similar to DARTS, mixed-weights on its own is also insufficient for searching over a diverse set of large kernels which is crucial to solving a wide range of problems.
Revised: However, we will show later that similar to DARTS, mixed-weights on its own is also insufﬁcient for searching over a diverse set of large kernels which is crucial to solving a wide range of problems.

Original: Expressive Diagonalization (XD)
Revised: Expressive Diagonalization (XD)

Original: Apart from linearity, XD-operations [7] propose to utilize the convolutional theorem in architecture search.
Revised: Apart from linearity, XD-operations [7] propose to utilize the convolutional theorem in architecture search.

Original: In particular, XD expresses the convolution acting on input x with filter w as K diag( Lw )
Revised: In particular, XD expresses the convolution acting on input x with ﬁlter w as K diag( Lw )

Original: Mx , where K , L , M are appropriate discrete Fourier transforms, and constructs an expansive search space by replacing these transforms with searched kaleidoscope matrices [29].
Revised: Mx , where K , L , M are appropriate discrete Fourier transforms, and constructs an expansive search space by replacing these transforms with searched kaleidoscope matrices [29].

Original: Although this new search space includes all types of convolutions, the search process is unacceptably long even for simple benchmarking tasks such as CIFAR-100 (Fig. 1b), let alone the more complex set of diverse problems that we consider in this paper.
Revised: Although this new search space includes all types of convolutions, the search process is unacceptably long even for simple benchmarking tasks such as CIFAR-100 (Fig. 1b), let alone the more complex set of diverse problems that we consider in this paper.

Original: In addition, the output architectures of XD are as inefficient as the supernet due to the absence of a discretization step.
Revised: In addition, the output architectures of XD are as inefﬁcient as the supernet due to the absence of a discretization step.

Original: Single-Path NAS
Revised: Single-Path NAS

Original: Lastly, Single-Path NAS [30] defines a large filter and uses its subsets for smaller filters.
Revised: Lastly, Single-Path NAS [30] deﬁnes a large ﬁlter and uses its subsets for smaller ﬁlters.

Original: This DARTS-based method compensates operator heterogeneity for efficiency during search.
Revised: This DARTS-based method compensates operator heterogeneity for efﬁciency during search.

Original: It does not handle search spaces with many large kernels and is not evaluated on diverse tasks.
Revised: It does not handle search spaces with many large kernels and is not evaluated on diverse tasks.

Original: Outside the field of AutoML, there is also emerging interest in designing general-purpose models such as Perceiver IO [31] and Frozen Pretrained Transformer [32].
Revised: Outside the ﬁeld of AutoML, there is also emerging interest in designing general-purpose models such as Perceiver IO [31] and Frozen Pretrained Transformer [32].

Original: However, these Transformer-based models do not adapt the network to the target tasks and are generally harder to train compared with CNNs.
Revised: However, these Transformer-based models do not adapt the network to the target tasks and are generally harder to train compared with CNNs.

Original: In Table 2, we evaluate Perceiver IO and show that its performance is not ideal.
Revised: In Table 2, we evaluate Perceiver IO and show that its performance is not ideal.

Revised: Recent works also propose to learn the kernel sizes for CNNs from the data [33], but they take an entirely different approach from NAS by multiplying the response of a convolution layer with a learnable Gaussian mask.

Original: Input: training data Z , loss function l , the set of kernel sizes K , the set of dilation rates D , and subsampling ratio p
Revised: Input: training data Z , loss function l , the set of kernel sizes K , the set of dilation rates D , and subsampling ratio p

Original: Initialize the backbone and replace each Conv layer with the mixed operation AggConv K,D while not converged do
Revised: Initialize the backbone and replace each Conv layer with the mixed operation AggConv K,D while not converged do

Original: Subsample p | Z | training points uniformly at random Compute forward pass using Equation
Revised: Subsample p | Z | training points uniformly at random Compute forward pass using Equation

Original: Descend the architecture parameters α by ∇ α l ( w , α ) and the model weights w by ∇ w l ( w , α ) end while
Revised: Descend the architecture parameters ↵ by r ↵ l ( w , ↵ ) and the model weights w by r w l ( w , ↵ ) end while

Original: Select arg max k ∈ K,d ∈ D α k,d for each AggConv layer Tune retraining hyperparameters on a validation subset of the training data
Revised: Select arg max k 2 K,d 2 D ↵ k,d for each AggConv layer Tune retraining hyperparameters on a validation subset of the training data

Original: Retrain the discretized model with all training data
Revised: Retrain the discretized model with all training data

Original:  Methods
Revised:  Methods

Original: Now, we describe the details of D ASH (Algorithm 1).
Revised: Now, we describe the details of D ASH (Algorithm 1).

Original: We first explain how D ASH leverages existing networks to initialize the supernet and generate different models for diverse tasks.
Revised: We ﬁrst explain how D ASH leverages existing networks to initialize the supernet and generate different models for diverse tasks.

Original: Then, we formally define the multi-scale convolution search space and propose a fast way to search this space using the three efficiency-motivated techniques mentioned earlier.
Revised: Then, we formally deﬁne the multi-scale convolution search space and propose a fast way to search this space using the three efﬁciency-motivated techniques mentioned earlier.

Original: Finally, we outline the procedure for discretizing the search space and retraining the searched model.
Revised: Finally, we outline the procedure for discretizing the search space and retraining the searched model.

Original: 3.1 Decoupling Topology and Operations
Revised: 3.1 Decoupling Topology and Operations

Original: Every architecture is a mapping from model weights to functions and can be described by a directed acyclic graph G ( V, E ) .
Revised: Every architecture is a mapping from model weights to functions and can be described by a directed acyclic graph G ( V, E ) .

Original: Each edge in E is characterized by ( u, v, Op ) , where u, v ∈ V are nodes and Op is an operation applied to u .
Revised: Each edge in E is characterized by ( u, v, Op ) , where u, v 2 V are nodes and Op is an operation applied to u .

Original: Node v aggregates the outputs of its incoming edges.
Revised: Node v aggregates the outputs of its incoming edges.

Original: NAS aims to automatically select the edge operations and the graph topology to optimize some objective.
Revised: NAS aims to automatically select the edge operations and the graph topology to optimize some objective.

Original: For each edge, Op is chosen from a search space S =
Revised: For each edge, Op is chosen from a search space S =

Original: { Op a | a ∈ A } where a ∈ A are architecture parameters.
Revised: { Op a | a 2 A } where a 2 A are architecture parameters.

Original: In past work, A usually indexes a small set of operations.
Revised: In past work, A usually indexes a small set of operations.

Original: For instance, the DARTS search space specifies A discrete = { 1 , . .
Revised: For instance, the DARTS search space speciﬁes A discrete = { 1 , . .

Original: , 8 } with S =
Revised: , 8 } with S =

Original: { Zero , Id , MaxPool 3 × 3 , AvgPool 3 × 3 , Conv 3 × 3 , Conv 5 × 5 , DilatedConv 3 × 3 , 2 , DilatedConv 5 × 5 , 2 } .
Revised: { Zero , Id , MaxPool 3 ⇥ 3 , AvgPool 3 ⇥ 3 , Conv 3 ⇥ 3 , Conv 5 ⇥ 5 , DilatedConv 3 ⇥ 3 , 2 , DilatedConv 5 ⇥ 5 , 2 } .

Original: However, A can also be defined systematically to identify operator properties, e.g., { ( kernel size k , dilation d ) } for convolutions.
Revised: However, A can also be deﬁned systematically to identify operator properties, e.g., { ( kernel size k , dilation d ) } for convolutions.

Original: A common way to determine the network topology is to search for blocks of operations and stack several blocks together.
Revised: A common way to determine the network topology is to search for blocks of operations and stack several blocks together.

Original: In this work, we take a different, morphism-based approach [23, 24, 7]: we use existing networks as backbones and replace certain layers in the backbone with the searched operations.
Revised: In this work, we take a different, morphism-based approach [23, 24, 7]: we use existing networks as backbones and replace certain layers in the backbone with the searched operations.

Original: Specifically, we select a set of architectures to accommodate both 2D and 1D datasets.
Revised: Speciﬁcally, we select a set of architectures to accommodate both 2D and 1D datasets.

Original: Convolutional layers with different kernels can then be plugged into these networks.
Revised: Convolutional layers with different kernels can then be plugged into these networks.

Original: An advantage of decoupling topology and operation search is flexibility: the searched operators can vary from the beginning to the end of a network, so features at different granularities can be processed differently.
Revised: An advantage of decoupling topology and operation search is ﬂexibility: the searched operators can vary from the beginning to the end of a network, so features at different granularities can be processed differently.

Original: We pick Wide ResNets (WRNs) [33, 34] as the backbone networks due to their simplicity and effectiveness in image and sequence modeling.
Revised: We pick Wide ResNets (WRNs) [34, 35] as the backbone networks due to their simplicity and effectiveness in image and sequence modeling.

Original: Before search, the supernet is initialized to the backbone.
Revised: Before search, the supernet is initialized to the backbone.

Original: Then, all Conv layers are substituted with an operator AggConv K,D (short for aggregated convolution) that represents the new search space which we now define.
Revised: Then, all Conv layers are substituted with an operator AggConv K,D (short for aggregated convolution) that represents the new search space which we now deﬁne.

Original: For simplicity, our mathematical discussion will stick to the 1D case, though our experiments are on both 1D and 2D data.
Revised: For simplicity, our mathematical discussion will stick to the 1D case, though our experiments are on both 1D and 2D data.

Original: 3.2 Efficiently Searching for Multi-Scale Convolutions
Revised: 3.2 Efﬁciently Searching for Multi-Scale Convolutions

Original: A convolution filter is specified by the kernel size k and the dilation rate
Revised: A convolution ﬁlter is speciﬁed by the kernel size k and the dilation rate

Original: d (we do not consider stride which does not change the filter shape).
Revised: d (we do not consider stride which does not change the ﬁlter shape).

Original: The effective filter size is ( k − 1) d
Revised: The effective ﬁlter size is ( k  1) d

Original: + 1 with nonzero entries separated by d − 1 zeros.
Revised: + 1 with nonzero entries separated by d  1 zeros.

Original: Let Conv k,d be the convolution with kernel size k , dilation rate d , c in input channels, and c out output channels.
Revised: Let Conv k,d be the convolution with kernel size k , dilation rate d , c in input channels, and c out output channels.

Original: Given input data with shape n , let K be our interested set of kernel sizes, D the set of dilations.
Revised: Given input data with shape n , let K be our interested set of kernel sizes, D the set of dilations.

Original: We define the AggConv K,D search space as
Revised: We deﬁne the AggConv K,D search space as

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Hence, A = K × D in previous notations.
Revised: Hence, A = K ⇥ D in previous notations.

Original: S AggConv K,D contains a collection of convolutions with receptive field size ranging from k min to d max ( k max − 1) + 1 , which allows us to discover models that process the input with varying resolution at each layer.
Revised: S AggConv K,D contains a collection of convolutions with receptive ﬁeld size ranging from k min to d max ( k max  1) + 1 , which allows us to discover models that process the input with varying resolution at each layer.

Original: To retain the efficiency of discrete NAS, we apply the continuous relaxation scheme of DARTS to S AggConv K,D , which mixes all operations in the space using architecture parameters { α k,d ∈ △ | K || D | | ( k, d ) ∈ K × D } 1 , so the output of each edge in the computational graph is 
Revised: To retain the efﬁciency of discrete NAS, we apply the continuous relaxation scheme of DARTS to S AggConv K,D , which mixes all operations in the space using architecture parameters { ↵ k,d4 | K || D | | ( k, d ) 2 K ⇥ D } 1 , so the output of each edge in the computational graph is [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here { w k,d | ( k, d ) ∈ K × D } are the kernel weights.
Revised: Here { w k,d | ( k, d ) 2 K ⇥ D } are the kernel weights.

Original: The resulting supernet can be trained end-toend, and our hope is that after search, the most important operation is assigned the highest weight.
Revised: The resulting supernet can be trained end-to- end, and our hope is that after search, the most important operation is assigned the highest weight.

Original: However, the complexity of computing the above summation directly, a baseline algorithmic approach we call mixed-results , is O ( c in c out ( | K || D | + ¯ K ) n ) , where ¯ K :=
Revised: However, the complexity of computing the above summation directly, a baseline algorithmic approach we call mixed-results , is O ( c in c out ( | K || D | + ¯ K ) n ) , where ¯ K :=

Original: | D | (cid:80) k ∈ K k .
Revised: | D | P k 2 K k .

Original: Mixed-results can be expensive when we increase the maximum element in K or D with larger input size n .
Revised: Mixed-results can be expensive when we increase the maximum element in K or D with larger input size n .

Original: To improve upon it, we propose three techniques which build up to the efficiency-oriented D ASH .
Revised: To improve upon it, we propose three techniques which build up to the efﬁciency-oriented D ASH .

Original: 3.2.1 Technique 1: Mixed-Weights
Revised: 3.2.1 Technique 1: Mixed-Weights

Original: Since convolution is linear, instead of computing | K || D | convolutions, we can combine the kernels and compute convolution once.
Revised: Since convolution is linear, instead of computing | K || D | convolutions, we can combine the kernels and compute convolution once.

Original: We call this approach mixed-weights :
Revised: We call this approach mixed-weights :

Original: [Equation]
Revised: [Equation]

Original: Here w ′ is the properly padded version of w (appending 0 ’s at the end of each dimension) so that filters of different sizes can be added.
Revised: Here w 0 is the properly padded version of w (appending 0 ’s at the end of each dimension) so that ﬁlters of different sizes can be added.

Original: The aggregated kernel has size ¯ D := max k,d ( k − 1) d + 1 and the n -dependent term of the complexity of mixed-weights is c in c out ¯ Dn .
Revised: The aggregated kernel has size ¯ D := max k,d ( k  1) d + 1 and the n -dependent term of the complexity of mixed-weights is c in c out ¯ Dn .

Original: Hence, it removes the direct dependence of the leading-order term on | K || D | , the search space size, that mixed-results had.
Revised: Hence, it removes the direct dependence of the leading-order term on | K || D | , the search space size, that mixed-results had.

Original: 3.2.2 Technique 2: Fourier Convolution
Revised: 3.2.2 Technique 2: Fourier Convolution

Original: If we wish to increase the kernel size and dilation with the input size, the complexity of mixed-weights will still grow implicitly with the search space size through the dependence on ¯ D .
Revised: If we wish to increase the kernel size and dilation with the input size, the complexity of mixed-weights will still grow implicitly with the search space size through the dependence on ¯ D .

Original: To address this issue, we combine the kernel re-weighting idea with another technique motivated by the convolution theorem.
Revised: To address this issue, we combine the kernel re-weighting idea with another technique motivated by the convolution theorem.

Original: Given a kernel w , recall that Conv ( w )( x ) =
Revised: Given a kernel w , recall that Conv ( w )( x ) =

Original: F − 1 diag( Fw ′ )
Revised: F  1 diag( Fw 0 )

Original: Fx , where F is the discrete
Revised: Fx , where F is the discrete

Original: Fourier transform (DFT) and diag( z ) is a diagonal matrix with entries z .
Revised: Fourier transform (DFT) and diag( z ) is a diagonal matrix with entries z .

Original: In other words, convolution involves multiplying the DFT of the kernel by that of the input.
Revised: In other words, convolution involves multiplying the DFT of the kernel by that of the input.

Original: Since the DFT can be applied in time
Revised: Since the DFT can be applied in time

Original: O ( n log n ) using the Fast Fourier Transform (FFT) and apart from that we only need element-wise multiplication, this yields an efficient approach to reducing dependence on the combined kernel size ¯
Revised: O ( n log n ) using the Fast Fourier Transform (FFT) and apart from that we only need element-wise multiplication, this yields an efﬁcient approach to reducing dependence on the combined kernel size ¯

Original: D .
Revised: D .

Original: While Roberts et al.
Revised: While Roberts et al.

Original: [7] replaced the DFTs with a continuous set of matrices for XD-operations, our approach can be viewed as replacing the middle DFT with a discrete set of matrices for the purpose of efficiency.
Revised: [7] replaced the DFTs with a continuous set of matrices for XD-operations, our approach can be viewed as replacing the middle DFT with a discrete set of matrices for the purpose of efﬁciency.

Original: Accordingly, D ASH computes AggConv K,D as follows:
Revised: Accordingly, D ASH computes AggConv K,D as follows:

Original: [Equation]
Revised: [Equation]

Original: Note that while the kernel changes for each Conv k,d , the input does not.
Revised: Note that while the kernel changes for each Conv k,d , the input does not.

Original: Hence, we also save time by transforming the input to the frequency domain only once.[Figure]
Revised: Hence, we also save time by transforming the input to the frequency domain only once.

Original: K,D .

Original: We vary the search space by letting K = { 2 p + 1 | 1 ≤ p ≤ c } , D = { 2 q − 1 | 1 ≤ q ≤ c } and increasing c from 1 to 7 .

Original: [Figure] In Table 1, we report the theoretical complexities of the baselines and D ASH , the latter leveraging both Technique 1 and 2.
Revised: In Table 1, we report the theoretical complexities of the baselines and D ASH , the latter leveraging both Technique 1 and 2.

Original: It is easy to obtain the operation complexities of mixed-weights and mixed-results .
Revised: It is easy to obtain the operation complexities of mixed-weights and mixed-results .

Original: For D ASH , the number of multiplications and additions can be attributed to the inner weight sum and multi-channel product (the first term) as well as three FFTs (the second).
Revised: For D ASH , the number of multiplications and additions can be attributed to the inner weight sum and multi-channel product (the ﬁrst term) as well as three FFTs (the second).

Original: A detailed analysis is provided in Appendix A.2.
Revised: A detailed analysis is provided in Appendix A.2.

Original: We see that mixed-weights is favorable to mixed-results when ¯
Revised: We see that mixed-weights is favorable to mixed-results when ¯

Original: D < ¯ K
Revised: D < ¯ K

Original: = O ( | D | k 2max ) , which occurs with large kernels and a few dilations.
Revised: = O ( | D | k 2max ) , which occurs with large kernels and a few dilations.

Original: On the other hand, only
Revised: On the other hand, only

Original: D ASH completely separates the dominant terms containing c in c out n from the size of the search space and its elements, replacing them by O (log n ) , which is small for any realistic n .
Revised: D ASH completely separates the dominant terms containing c in c out n from the size of the search space and its elements, replacing them by O (log n ) , which is small for any realistic n .

Original: As we increase k max and d max for larger inputs, this will also lead to a slower asymptotic increase in complexity, making D ASH an attractive choice for the multi-scale search space where ¯ D is large by design to extract possible long-range dependencies in the data.
Revised: As we increase k max and d max for larger inputs, this will also lead to a slower asymptotic increase in complexity, making D ASH an attractive choice for the multi-scale search space where ¯ D is large by design to extract possible long-range dependencies in the data.

Original: 3.2.3 Technique 3: Kronecker Dilation
Revised: 3.2.3 Technique 3: Kronecker Dilation

Original: To efficiently implement the kernel summation in mixed-weights and D ASH on a GPU, we introduce our final technique: after initializing w k,d for each Conv k,d separately, we use a Kronecker product ⊗ to transform the undilated kernels into dilated forms.
Revised: To efﬁciently implement the kernel summation in mixed-weights and D ASH on a GPU, we introduce our ﬁnal technique: after initializing w k,d for each Conv k,d separately, we use a Kronecker product ⌦ to transform the undilated kernels into dilated forms.

Original: For example, to compute a 2D convolution with dilation d , we introduce the sparse pattern matrix P ∈ R d × d whose entries are all 0 ’s except for the upper-left entry P 1 , 1 = 1 :[Equation]
Revised: For example, to compute a 2D convolution with dilation d , we introduce the sparse pattern matrix P 2 R d ⇥ d whose entries are all 0 ’s except for the upper-left entry P 1 , 1 = 1 :

Original: [Equation]
Revised: [Equation]

Original: Then, w k,d = w k, 1 ⊗ P .
Revised: Then, w k,d = w k, 1 ⌦ P .

Original: Beyond the theoretical gains shown in Wu et al.
Revised: Beyond the theoretical gains shown in Wu et al.

Original: [35], this dilation strategy is empirically faster than the standard way of padding 0 ’s into w k, 1 (Fig. 2 and 3).
Revised: [36], this dilation strategy is empirically faster than the standard way of padding 0 ’s into w k, 1 (Fig. 2 and 3).

Original: After dilating the kernels, we sum them together, zero-pad to match the input size, and apply the FFTs.
Revised: After dilating the kernels, we sum them together, zero-pad to match the input size, and apply the FFTs.

Original: 3.2.4 Ablation Study for the Proposed Techniques
Revised: 3.2.4 Ablation Study for the Proposed Techniques

Original: To check that our asymptotic analysis leads to actual speedups and perform an ablation study on the proposed techniques, we evaluate the three methods on single-channel 1D input (experiment details are in Appendix A.3).
Revised: To check that our asymptotic analysis leads to actual speedups and perform an ablation study on the proposed techniques, we evaluate the three methods on single-channel 1D input (experiment details are in Appendix A.3).

Original: Since both mixed-weights and D ASH require kernel summation, which can be implemented with Kronecker dilation (Technique 3), we compare five methods in total.
Revised: Since both mixed-weights and D ASH require kernel summation, which can be implemented with Kronecker dilation (Technique 3), we compare ﬁve methods in total.

Original: Fig.
Revised: Fig.

Original: 2 illustrates the combined forward- and backward-pass time in log scale for one search epoch vs. the size of S AggConv K,D when n = 1000 .
Revised: 2 illustrates the combined forward- and backward-pass time in log scale for one search epoch vs. the size of S AggConv K,D when n = 1000 .

Original: For small ¯ D , the FFT overhead makes D ASH runtime slightly longer but the difference is negligible.
Revised: For small ¯ D , the FFT overhead makes D ASH runtime slightly longer but the difference is negligible.

Original: However, as ¯ D increases, the D ASH curves grow much slower whereas the runtimes for the other methods scale with the number of operations.
Revised: However, as ¯ D increases, the D ASH curves grow much slower whereas the runtimes for the other methods scale with the number of operations.

Original: In Fig.
Revised: In Fig.

Original: 3, we fix K and D to study how runtime is affected by input size n .
Revised: 3, we ﬁx K and D to study how runtime is affected by input size n .

Original: Both mixed-results and mixed-weights become extremely inefficient for large n ’s which commonly occur in time-series or signal processing.
Revised: Both mixed-results and mixed-weights become extremely inefﬁcient for large n ’s which commonly occur in time-series or signal processing.

Original: Surprisingly, D ASH ’s runtime does not increase much with n .
Revised: Surprisingly, D ASH ’s runtime does not increase much with n .

Original: We hypothesize that this is due to wallclock-time being dominated by data-passing at that speed.
Revised: We hypothesize that this is due to wallclock-time being dominated by data-passing at that speed.

Original: In general, Technique 1 on its own scales poorly for the considered search space.
Revised: In general, Technique 1 on its own scales poorly for the considered search space.

Original: This is why methods like MergeNAS [27] cannot be used in our setting.
Revised: This is why methods like MergeNAS [27] cannot be used in our setting.

Original: Though XD makes use of Technique 2, it considers a parametrized space with infinitely many operations that need to be continuously evolved and is too expensive to be applied to tasks beyond CIFAR-100 (c.f. Fig. 1b).
Revised: Though XD makes use of Technique 2, it considers a parametrized space with inﬁnitely many operations that need to be continuously evolved and is too expensive to be applied to tasks beyond CIFAR-100 (c.f. Fig. 1b).

Original: Technique 3 contributes to 2 × speedups for both mixed-weights and D ASH .
Revised: Technique 3 contributes to 2 ⇥ speedups for both mixed-weights and D ASH .

Original: Overall, D ASH (Kronecker) leads to about 10 × search-time speedups compared to the mixed-results scheme of DARTS for both the large operation space and large input size regimes.
Revised: Overall, D ASH (Kronecker) leads to about 10 ⇥ search-time speedups compared to the mixed-results scheme of DARTS for both the large operation space and large input size regimes.

Original: Hence, we use this version of D ASH in later experiments.
Revised: Hence, we use this version of D ASH in later experiments.

Original: 3.3 The Full Pipeline: Architecture Search, Hyperparameter Optimization, and Retraining
Revised: 3.3 The Full Pipeline: Architecture Search, Hyperparameter Optimization, and Retraining

Original: Having shown the main techniques for searching a large space of kernel patterns, we now detail the full search and model development pipeline.
Revised: Having shown the main techniques for searching a large space of kernel patterns, we now detail the full search and model development pipeline.

Original: Given a dataset, we set K = {
Revised: Given a dataset, we set K = {

Original: 3 + d ( p − 1) | 1 ≤ p ≤ p max } for kernel sizes and D = { 2 q − 1 | 1 ≤ q ≤ q max } for dilations.
Revised: 3 + d ( p  1) | 1  p  p max } for kernel sizes and D = { 2 q  1 | 1  q  q max } for dilations.

Original: For 2D input, we set d to 2, p max to 4 , and q max to 4.
Revised: For 2D input, we set d to 2, p max to 4 , and q max to 4.

Original: For longer 1D sequence data, we set d = 4 , p max = 5 , and q max = 4 .
Revised: For longer 1D sequence data, we set d = 4 , p max = 5 , and q max = 4 .

Original: For instance,
Revised: For instance,

Original: CIFAR has size 3 × 32 × 32 where 3 is the number of channels and n = 32 .
Revised: CIFAR has size 3 ⇥ 32 ⇥ 32 where 3 is the number of channels and n = 32 .

Original: The corresponding K is { 3 , 5 , 7 , 9 } and D is { 1 , 3 , 7 , 15 } .
Revised: The corresponding K is { 3 , 5 , 7 , 9 } and D is { 1 , 3 , 7 , 15 } .

Original: To normalize architecture parameters into a probability distribution, we adopt the soft Gumbel Softmax activation, similar to Xie et al.
Revised: To normalize architecture parameters into a probability distribution, we adopt the soft Gumbel Softmax activation, similar to Xie et al.

Original: The backbone networks are as follows.
Revised: The backbone networks are as follows.

Original: For 2D tasks, we use WRN 16-1 as the search backbone to accelerate supernet training and WRN 16-4 for retraining.
Revised: For 2D tasks, we use WRN 16-1 as the search backbone to accelerate supernet training and WRN 16-4 for retraining.

Original: For 1D tasks, we use 1D WRN [34] in the entire pipeline.
Revised: For 1D tasks, we use 1D WRN [35] in the entire pipeline.

Original: During search, we subsample the training data at each epoch.
Revised: During search, we subsample the training data at each epoch.

Original: Given the loss for the target task, D ASH jointly optimizes the model weights and the architecture parameters using direct gradient descent.
Revised: Given the loss for the target task, D ASH jointly optimizes the model weights and the architecture parameters using direct gradient descent.

Original: This single-level optimization is more efficient than two-stage NAS, which finds initial assignments for architecture parameters and trains the candidates from scratch.
Revised: This single-level optimization is more efﬁcient than two-stage NAS, which ﬁnds initial assignments for architecture parameters and trains the candidates from scratch.

Original: After searching for a predefined number of epochs, we discretize the search space and pick Conv k,d∈
Revised: After searching for a predeﬁned number of epochs, we discretize the search space and pick Conv k,d

Original: S AggConv K,D with the largest weight for each layer.
Revised: S AggConv K,D with the largest weight for each layer.

Original: The final model has a similar overall structure to the backbone, but the intermediate operations are tailored to the target task.
Revised: The ﬁnal model has a similar overall structure to the backbone, but the intermediate operations are tailored to the target task.

Original: To improve training stability, we additionally add a simple hyperparameter tuning stage between search and retraining using grid search (configuration space shown in Appendix A.5.2).
Revised: To improve training stability, we additionally add a simple hyperparameter tuning stage between search and retraining using grid search (conﬁguration space shown in Appendix A.5.2).

Original: For each setting, we train the discretized model on a subset of the training data for fewer epochs so the tuning cost is a small fraction of the entire pipeline’s cost (Table 6), Then, we evaluate the performance on a holdout validation set and select the configuration with the best validation score.
Revised: For each setting, we train the discretized model on a subset of the training data for fewer epochs so the tuning cost is a small fraction of the entire pipeline’s cost (Table 6), Then, we evaluate the performance on a holdout validation set and select the conﬁguration with the best validation score.

Original: As a final step, we retrain the discretized model with the optimal hyperparameters on all training data until convergence.
Revised: As a ﬁnal step, we retrain the discretized model with the optimal hyperparameters on all training data until convergence.

Original: Like other weight-sharing methods with discretization, our final model will be more efficient than the supernet.
Revised: Like other weight-sharing methods with discretization, our ﬁnal model will be more efﬁcient than the supernet.

Original:  Evaluation
Revised:  Evaluation

Original: We evaluate the performance of D ASH on diverse tasks using ten datasets from NAS-Bench-360 [4], a benchmark spanning multiple application domains, input dimensions, and learning objectives.
Revised: We evaluate the performance of D ASH on diverse tasks using ten datasets from NAS-Bench-360 [4], a benchmark spanning multiple application domains, input dimensions, and learning objectives.

Original: These include classical vision tasks such as CIFAR-100 where CNNs do well, scientific computing tasks such as Darcy Flow where standard CNN backbones can perform poorly [7, 36], sequence tasks such as DeepSEA where large dilations are preferred [13, 26], and many others.
Revised: These include classical vision tasks such as CIFAR-100 where CNNs do well, scientiﬁc computing tasks such as Darcy Flow where standard CNN backbones can perform poorly [7, 37], sequence tasks such as DeepSEA where large dilations are preferred [13, 26], and many others.

Original: Thus, our evaluation will not only test whether D ASH can find good architectures in the proposed new search space, but also investigate whether multi-scale convolution is a strong competitor for solving different problems.
Revised: Thus, our evaluation will not only test whether D ASH can ﬁnd good architectures in the proposed new search space, but also investigate whether multi-scale convolution is a strong competitor for solving different problems.

Original: In fact, our results show that D ASH is a top choice for many tasks, obtaining in-aggregate the best speed-accuracy trade-offs among the methods we evaluate (c.f. Fig. 5).
Revised: In fact, our results show that D ASH is a top choice for many tasks, obtaining in-aggregate the best speed-accuracy trade-offs among the methods we evaluate (c.f. Fig. 5).

Original: 4.1 Baselines and Experimental Setup
Revised: 4.1 Baselines and Experimental Setup

Original: For each NAS-Bench-360 task, we compare D ASH with the following methods: DenseNAS [5] and GAEA PC-DARTS [3], which represent general NAS; Auto-DeepLab [37] and AMBER [26], which represent specialist NAS methods for dense prediction and 1D tasks, respectively; 1D temporal convolutional network (TCN) [13], regular WRN, and WRN with hyperparameter tuner ASHA [38], which represent natural NAS baselines; and Perceiver IO [31], which represents non-NAS generalpurpose models. While these results are available in Tu et al.
Revised: For each NAS-Bench-360 task, we compare D ASH with the following methods: DenseNAS [5] and GAEA PC-DARTS [3], which represent general NAS; Auto-DeepLab [38] and AMBER [26], which represent specialist NAS methods for dense prediction and 1D tasks, respectively; 1D temporal convolutional network (TCN) [13], regular WRN, and WRN with hyperparameter tuner ASHA [39], which represent natural NAS baselines; and Perceiver IO [31], which represents non-NAS generalpurpose models these results , we

Original: [4], we additionally add a BABY D ASH 2 For completeness, we give a task summary in the Appendix.
Revised: B

Original: [Table]
Revised: [Table]

Original: baseline: we run D ASH in the DARTS convolution space with K = { 3 , 5 } and D = { 1 , 2 } to study whether large kernel sizes and dilations are necessary to strong performance across-the-board.
Revised: baseline: we run D ASH in the DARTS convolution space with K = { 3 , 5 } and D = { 1 , 2 } to study whether large kernel sizes and dilations are necessary to strong performance across-the-board.

Original: Finally, we compare our method to the expert architectures selected by NAS-Bench-360.
Revised: Finally, we compare our method to the expert architectures selected by NAS-Bench-360.

Original: These models are representatives of the best that hand-crafting has to offer.
Revised: These models are representatives of the best that hand-crafting has to offer.

Original: Each dataset is preprocessed and split using the NAS-Bench-360 script, with the training set being used for search, hyperparameter tuning, and retraining.
Revised: Each dataset is preprocessed and split using the NAS-Bench-360 script, with the training set being used for search, hyperparameter tuning, and retraining.

Original: To construct the multi-scale search space, we set K and D according to the rules in Section 3.3.
Revised: To construct the multi-scale search space, we set K and D according to the rules in Section 3.3.

Original: We use the default SGD optimizer for the WRN backbone and fix the learning rate schedule as well as the gradient clipping threshold for every task.
Revised: We use the default SGD optimizer for the WRN backbone and ﬁx the learning rate schedule as well as the gradient clipping threshold for every task.

Original: The entire D ASH pipeline can be run on a single NVIDIA V100 GPU, which is also the system that we use to report the runtime cost.
Revised: The entire D ASH pipeline can be run on a single NVIDIA V100 GPU, which is also the system that we use to report the runtime cost.

Original: Full experimental details can be found in the Appendix.
Revised: Full experimental details can be found in the Appendix.

Original: We evaluate the performance of all competing methods following the NAS-Bench-360 protocol.
Revised: We evaluate the performance of all competing methods following the NAS-Bench-360 protocol.

Original: We first report results of the target metric for each task by running the model of the last epoch on the test data.
Revised: We ﬁrst report results of the target metric for each task by running the model of the last epoch on the test data.

Original: Then, we report aggregate results via performance profiles [39], a technique that considers both outliers and small performance differences to compare methods across multiple tasks robustly.
Revised: Then, we report aggregate results via performance proﬁles [40], a technique that considers both outliers and small performance differences to compare methods across multiple tasks robustly.

Original: In such plots, each curve represents one method.
Revised: In such plots, each curve represents one method.

Original: The τ on the x -axis denotes the fraction of tasks on which a method is no worse than a τ -factor from the best.
Revised: The ⌧ on the x -axis denotes the fraction of tasks on which a method is no worse than a ⌧ -factor from the best.

Original: 4.2 Results and Discussion
Revised: 4.2 Results and Discussion

Original: We present the accuracy results for each task in Table 2 and the performance profiles in Fig.
Revised: We present the accuracy results for each task in Table 2 and the performance proﬁles in Fig.

Original: Fig. 4 clearly demonstrates that D ASH is superior to other competing methods in terms of aggregate performance.
Revised: Fig. 4 clearly demonstrates that D ASH is superior to other competing methods in terms of aggregate performance.

Original: In particular, it ranks first among all automated models for 7/10 tasks, among all expert models for 7/10 tasks, and performs favorably when considering both accuracy and efficiency as shown in Fig.
Revised: In particular, it ranks ﬁrst among all automated models for 7/10 tasks, among all expert models for 7/10 tasks, and performs favorably when considering both accuracy and efﬁciency as shown in Fig.

Original: In addition, Table 3 shows that D ASH outperforms DARTS in speed for all 10 tasks (in several cases by an order of magnitude), and attains comparable efficiency with training vanilla
Revised: In addition, Table 3 shows that D ASH outperforms DARTS in speed for all 10 tasks (in several cases by an order of magnitude), and attains comparable efﬁciency with training vanilla

Original: WRNs for 6/10 tasks (full-pipeline time is less than or about twice as long as the WRN training time).
Revised: WRNs for 6/10 tasks (full-pipeline time is less than or about twice as long as the WRN training time).

Original: In the following, we provide a detailed analysis of the experimental results.
Revised: In the following, we provide a detailed analysis of the experimental results.

Original: D ASH dominates automated methods.
Revised: D ASH dominates automated methods.

Original: Compared to other automated methods, D ASH has a clear advantage in accuracy.
Revised: Compared to other automated methods, D ASH has a clear advantage in accuracy.

Original: Even for tasks where it does not beat the expert, e.g. ECG, D ASH ’s performance is significantly better than other AutoML methods.
Revised: Even for tasks where it does not beat the expert, e.g. ECG, D ASH ’s performance is signiﬁcantly better than other AutoML methods.

Original: It also outperforms specialist methods Auto-DL and AMBER on dense prediction and 1D tasks, respectively.
Revised: It also outperforms specialist methods Auto-DL and AMBER on dense prediction and 1D tasks, respectively.

Original: Although DARTS does best on CIFAR-100 (the task for which it was designed), Spherical, and PSICOV, it is the worst on NinaPro and FSD50K. Note that the underperformance of D ASH on CIFAR-100 relative to WRN (and on Spherical and Cosmic relative to B ABY D ASH ) suggests suboptimality of the gradient descent optimization procedure but not of the operation space, since WRN and B ABY D ASH are contained in our search space.
Revised: Although DARTS does best on CIFAR-100 (the task for which it was designed), Spherical, and PSICOV, it is the worst on NinaPro and FSD50K. Note that the underperformance of D ASH on CIFAR-100 relative to WRN (and on Spherical and Cosmic relative to B ABY D ASH ) suggests suboptimality of the gradient descent optimization procedure but not of the operation space, since WRN and B ABY D ASH are contained in our search space.

Original: This indicates a future direction to improve optimization in the D ASH search space.[Figure] [Figure]
Revised: This indicates a future direction to improve optimization in the D ASH search space.

Original: D ASH dominates expert architectures.
Revised: D ASH dominates expert architectures.

Original: While the degree of sophistication of the expert networks varies task by task, the performance of D ASH on tasks such as Darcy Flow suggests that it is capable of competing with highly specialized networks, e  g., Fourier Neural Operator [36] for PDE solving.
Revised: While the degree of sophistication of the expert networks varies task by task, the performance of D ASH on tasks such as Darcy Flow suggests that it is capable of competing with highly specialized networks, e [Table] g., Fourier Neural Operator [37] for PDE solving.

Original: These results imply that D ASH , and more generally the strategy of equipping backbone networks with task-specific kernels is a promising approach for tackling model development in new domains.
Revised: These results imply that D ASH , and more generally the strategy of equipping backbone networks with task-speciﬁc kernels, is a promising approach for tackling model development in new domains.

Original: Meanwhile, D ASH consistently outperforms Perceiver IO which represents non-automated generalpurpose models.
Revised: Meanwhile, D ASH consistently outperforms Perceiver IO which represents non-automated generalpurpose models.

Original: We speculate that the poor performance of Perceiver IO is because it is developed on more sophisticated natural language and multi-modal reasoning tasks and training Transformers is generally difficult simply training is its accuracy is far less is too expensive to be applied
Revised: We speculate that the poor performance of Perceiver IO is because it is developed on more sophisticated natural language and multi-modal reasoning tasks and training Transformers is generally difﬁcult [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Large kernels are needed.
Revised: Large kernels are needed.

Original: We also ablate the large- k -large- d design of the search space by comparing D ASH with B ABY D ASH .
Revised: We also ablate the large- k -large- d design of the search space by comparing D ASH with B ABY D ASH .

Original: We hypothesize that for the same task, a small performance gap between the two methods would indicate that small kernels suffice for extracting local features, whereas a major degradation in the quality of the
Revised: We hypothesize that for the same task, a small performance gap between the two methods would indicate that small kernels sufﬁce for extracting local features, whereas a major degradation in the quality of the

Original: B ABY D ASH model can imply that the task needs global modeling.
Revised: B ABY D ASH model can imply that the task needs global modeling.

Original: Consequently, datasets such as
Revised: Consequently, datasets such as

Original: Darcy Flow and ECG provide compelling evidence that kernels with large receptive fields play an important role in solving real-life problems and further back up the design of our multi-scale convolutional search space.
Revised: Darcy Flow and ECG provide compelling evidence that kernels with large receptive ﬁelds play an important role in solving real-life problems and further back up the design of our multi-scale convolutional search space.

Original: An example of the series of WRN kernels found by D ASH on Darcy
Revised: An example of the series of WRN kernels found by D ASH on Darcy

Original: Flow is:
Revised: Flow is:

Original: [Equation]
Revised: [Equation]

Original: We can see that large kernels are indeed selected during search.
Revised: We can see that large kernels are indeed selected during search.

Original: More visualizations can be found in Appendix A.8.
Revised: More visualizations can be found in Appendix A.8.

Original:  ASH is computationally efficient.
Revised: D ASH is computationally efﬁcient.

Original: In addition to prediction quality, we also care about the efficiency of model selection.
Revised: In addition to prediction quality, we also care about the efﬁciency of model selection.

Original: Table 3 provides the combined search and retraining time in GPU hours for DARTS, DenseNAS, and D ASH , as well as the training time for vanilla WRN 16-4 without hyperparameter tuning (baseline results are taken from Tu et al.
Revised: Table 3 provides the combined search and retraining time in GPU hours for DARTS, DenseNAS, and D ASH , as well as the training time for vanilla WRN 16-4 without hyperparameter tuning (baseline results are taken from Tu et al.

Original: We also present the breakdown of D ASH ’s full-pipeline runtime in Appendix A.7.
Revised: We also present the breakdown of D ASH ’s full-pipeline runtime in Appendix A.7.

Original: A key observation is that the cost of D ASH is consistently below DARTS’ on all tasks and is similar to training a simple CNN for more than half of them.
Revised: A key observation is that the cost of D ASH is consistently below DARTS’ on all tasks and is similar to training a simple CNN for more than half of them.

Original: Although DenseNAS stands out by speed, its practical performance is less impressive.
Revised: Although DenseNAS stands out by speed, its practical performance is less impressive.

Original: In Fig.
Revised: In Fig.

Original: 5, we visualize the trade-off between efficiency and effectiveness for each method and task combination.
Revised: 5, we visualize the trade-off between efﬁciency and effectiveness for each method and task combination.

Original: Evidently, D ASH takes an important step towards bridging the gap between the efficiency of DARTS and the expressivity of XD in NAS.
Revised: Evidently, D ASH takes an important step towards bridging the gap between the efﬁciency of DARTS and the expressivity of XD in NAS.

Original: The fact that D ASH can be trained at a low cost testifies that we need not sacrifice efficiency for adding more operations.
Revised: The fact that D ASH can be trained at a low cost testiﬁes that we need not sacriﬁce efﬁciency for adding more operations.

Original: In fact, we have actually shown that D ASH is both faster and more effective than DARTS in many tasks.
Revised: In fact, we have actually shown that D ASH is both faster and more effective than DARTS in many tasks.

Original: D ASH works with other backbones and is backward compatible.
Revised: D ASH works with other backbones and is backward compatible.

Original: Lastly, in addition to the Wide
Revised: Lastly, in addition to the Wide

Original: ResNet backbone and NAS-Bench-360 tasks, we have also verified the efficacy of D ASH on other backbones including TCN [13] and ConvNeXt [15], and on large-scale datasets including ImageNet in Appendix A.9.
Revised: ResNet backbone and NAS-Bench-360 tasks, we have also veriﬁed the efﬁcacy of D ASH on other backbones including TCN [13] and ConvNeXt [15], and on large-scale datasets including ImageNet in Appendix A.9.

Original: In particular, D ASH is able to achieve a 1.4% increase in top-1 accuracy for
Revised: In particular, D ASH is able to achieve a 1.4% increase in top-1 accuracy for

Original: ImageNet on top of the ConvNeXt backbone.
Revised: ImageNet on top of the ConvNeXt backbone.

Original: As the latter was itself developed in part via manual tuning of the kernel size, this means that D ASH outperforms human hand-tuning on ImageNet.
Revised: As the latter was itself developed in part via manual tuning of the kernel size, this means that D ASH outperforms human hand-tuning on ImageNet.

Original: These results show that D ASH is backbone-agnostic and also works well with computer vision tasks, making it backward compatible with the original use cases of CNNs.
Revised: These results show that D ASH is backbone-agnostic and also works well with computer vision tasks, making it backward compatible with the original use cases of CNNs.

Original: 4.3 Limitations and Future Work
Revised: 4.3 Limitations and Future Work

Original: There are several open problems which we leave for future work.
Revised: There are several open problems which we leave for future work.

Original: First, it is beneficial to study why certain kernel patterns are chosen, as the selected operations can hint us at the intrinsic properties of the datasets.
Revised: First, it is beneﬁcial to study why certain kernel patterns are chosen, as the selected operations can hint us at the intrinsic properties of the datasets.

Original: Second, one can improve upon D ASH , e.g., by including non-square convolutions for 2D problems, using a better optimization algorithm, or developing techniques that further reduces the memory usage of performing a forward architecture search pass.
Revised: Second, one can improve upon D ASH , e.g., by including non-square convolutions for 2D problems, using a better optimization algorithm, or developing techniques that further reduces the memory usage of performing a forward architecture search pass.

Original: One could also construct a more comprehensive search space with high-level operators such as self-attention [40].
Revised: One could also construct a more comprehensive search space with high-level operators such as self-attention [41].

Original: Meanwhile, although this paper focuses on NAS, which is an alternative to fine-tuning pretrained models, the aggregated convolution can be a plug-and-play module for algorithms that search for large-scale models.
Revised: Meanwhile, although this paper focuses on NAS, which is an alternative to ﬁne-tuning pretrained models, the aggregated convolution can be a plug-and-play module for algorithms that search for large-scale models.

Original: For instance, many Transformer models still depend on convolutions for feature extraction and transformation, and their performance relies on the quality of the embedded features.
Revised: For instance, many Transformer models still depend on convolutions for feature extraction and transformation, and their performance relies on the quality of the embedded features.

Original: Since D ASH is applicable to any architecture with a convolutional layer, it can be helpful for such models, including Vision Transformer with a convolutional patching layer [41], Deformable
Revised: Since D ASH is applicable to any architecture with a convolutional layer, it can be helpful for such models, including Vision Transformer with a convolutional patching layer [42], Deformable

Original: Transformer with a ResNet embedder [42], Swin Transformer with a convolutional decoder [43], and many others.
Revised: Transformer with a ResNet embedder [43], Swin Transformer with a convolutional decoder [44], and many others.

Original: Societal Impact
Revised: Societal Impact

Original: This paper calls for ML community’s attention to less-studied application domains and moves towards truly democratizing machine learning in real life.
Revised: This paper calls for ML community’s attention to less-studied application domains and moves towards truly democratizing machine learning in real life.

Original: In terms of broader societal impact, our work can exert a positive influence as it contributes to NAS efficiency and reduces the computational burden on AutoML end-users.
Revised: In terms of broader societal impact, our work can exert a positive inﬂuence as it contributes to NAS efﬁciency and reduces the computational burden on AutoML end-users.

Original: However, lowering the barrier for applying ML to a wide range of tasks necessarily comes with the risk of misuse.
Revised: However, lowering the barrier for applying ML to a wide range of tasks necessarily comes with the risk of misuse.

Original: Hence, it is imperative to develop
Revised: Hence, it is imperative to develop

Original: NAS methods with privacy, safety, and fairness guarantees.
Revised: NAS methods with privacy, safety, and fairness guarantees.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we argue that a crucial goal of NAS is to discover accessible models for diverse tasks.
Revised: In this paper, we argue that a crucial goal of NAS is to discover accessible models for diverse tasks.

Original: To this end, we propose D ASH , which efficiently searches for convolution patterns and integrates them into existing backbones.
Revised: To this end, we propose D ASH , which efﬁciently searches for convolution patterns and integrates them into existing backbones.

Original: D ASH overcomes the computational limitations of differentiable NAS and obtains high-quality models with accuracy comparable to or better than that of the handcrafted networks on many tasks.
Revised: D ASH overcomes the computational limitations of differentiable NAS and obtains high-quality models with accuracy comparable to or better than that of the handcrafted networks on many tasks.

Original: Our experiments show that convolution can be a universal operator for many under-explored areas.
Revised: Our experiments show that convolution can be a universal operator for many under-explored areas.

Original: D ASH is also a promising step towards developing general-purpose models with more complicated structures.
Revised: D ASH is also a promising step towards developing general-purpose models with more complicated structures.

Original: Acknowledgments
Revised: Acknowledgments

Original: We thank Maria-Florina Balcan, Nicholas Roberts, and Renbo Tu for providing useful feedback.
Revised: We thank Maria-Florina Balcan, Nicholas Roberts, and Renbo Tu for providing useful feedback.

Original: This work was supported in part by DARPA FA875017C0141, the National Science Foundation grants IIS1705121, IIS1838017, IIS2046613 and IIS-2112471, an Amazon Web Services Award, a
Revised: This work was supported in part by DARPA FA875017C0141, the National Science Foundation grants IIS1705121, IIS1838017, IIS2046613 and IIS-2112471, an Amazon Web Services Award, a

Original: Facebook Faculty Research Award, funding from Booz Allen Hamilton Inc., a Block Center Grant, and a Facebook Fellowship Award.
Revised: Facebook Faculty Research Award, funding from Booz Allen Hamilton Inc., a Block Center Grant, and a Facebook Fellowship Award.

Original: Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of any of these funding agencies.A Appendix A.1 Term Clarification
Revised: Any opinions, ﬁndings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reﬂect the views of any of these funding agencies.

Original: Since we compare with a variety of methods in the paper, here we clarify some of the terms we use.

Original: what we say what we are referring to

Original: Best-Automated (Fig. 1a)

Original: WRN, WRN-ASHA, DARTS, DenseNAS, Auto-DL, AMBER

Original: Hand-Designed (Fig. 1a)

Original: Expert architectures in Table

Original: AutoML

Original: WRN-ASHA, DARTS, DenseNAS, Auto-DL, AMBER

Original: NAS

Original: DARTS, DenseNAS, Auto-DL, AMBER

Original: WRN

Original: WRN without hyperparameter tuning

Original: A.2 Asymptotic Analysis

Original: In this section we outline the runtime analysis used to populate the asymptotic complexities in

Original: Table 1.

Original: All three methods in the table— mixed-results , mixed-weights , and D ASH —are computing the following weighted sum of convolutions:

Original: [Equation]

Original: We consider 1D inputs x with length n and c in input channels; the convolutions have c out output channels.

Original: We view Conv ( w k,d )( x ) as having the naive complexity c in c out kn since the deep learning frameworks use the direct (non-Fourier) algorithm.

Original: mixed-results computes the sum directly, which involves (1) applying one convolution of each size k and dilation to x at a cost of c in c out kn

Original: MULTs and ADDs each for a total cost of c in c out

Original: ¯

Original: Kn , (2) scalar-multiplying the outputs at a cost of c out | K || D | n MULTs, and (3) summing the results together at a cost of c out | K || D | n ADDs.

Original: mixed-weights instead (1) multiplies all kernels by their corresponding weight at a cost of c in c out

Original: ¯ K

Original: MULTs, (2) zero-pads the results to the largest effective kernel size ¯ D and adds them together at a cost of c in c out | K || D | ¯ D ADDs, and (3) applies the resulting ¯ D -size convolution to the input at a cost of c in c out ¯ Dn MULTs and ADDs.

Original: Finally, D ASH also (1) does the first two steps of mixed-weights at a cost of c in c out

Original: ¯ K MULTs and c in c out | K || D | ¯ D ADDs but then (2) pads the resulting ¯ D -size convolution to size n and applies an FFT at a cost of O ( c in c out n log n ) MULTs and ADDs, (3) applies an FFT to x at a cost of O ( c in n log n ) , (4) element-wise multiplies the transformed filters by the inputs at a cost of c in c out n MULTs, (5) adds up c in results for each of c out output channels at a cost of c in c out MULTs, and (6) applies an iFFT to the result at a cost of O ( c out n log n ) .

Original: A.3 Experiment Details for

Original: Fig. 2 and Fig.

Original: For the speed tests, we work with the Sequential MNIST dataset, i.e., the 2D 28 × 28 images are stretched into 1D with length 784 .

Original: We zero pad or truncate the input to generate data with different input size n .

Original: The backbone is 1D WRN with the same structure as introduced in Section 3.

Original: The batch size is 128 .

Original: We run the workflow on a single NVIDIA V100 GPU.

Original: The timing results reported are the log 10 ( combined forward and backward pass time for one search epoch ) .

Original: In Fig.

Original: 2, we study how the size of our multi-scale convolution search space affects the runtimes of mixed-results , mixed-weights , and D ASH for n = 1000 (zero-padded MNIST).

Original: We define K = { 3 + 2( p − 1)

Original: | 1 ≤ p ≤ c } , D = { 2 q − 1 | 1 ≤ q ≤ c } and varies c from 1 to 7 .

Original: Consequently, the number of operations included in the search space grows from 1 to 49 .

Original: In Fig.

Original: 3, we study how the input size affects the runtimes of the three methods.

Original: We fix K = { 3 , 5 , 7 , 9 , 11 } , D = { 1 , 3 , 7 , 15 , 31 } and vary n from 2 5 to 2 12 .

Original: A.4 Information About

Original: in NAS-Bench-

Original: [Table]

Original: A.5 Evaluation of D ASH on NAS-BenchA.5.1 Backbone Network Structure

Original: 2D Tasks We use the Wide ResNet 16-4 [33] as the backbone for all 2D tasks.

Original: The original model is made up of 16 3 × 3 conv followed by 6 WRN blocks with the following structure ( i ∈ { 1 , 2 , 3 , 4 , 5 , 6 } indicates the block index):

Original: BatchNorm, ReLU

Original: Conv16 × 4 × (( i + 1) // 2) ( k = 3 , d = 1) filters, ReLU

Original: Dropout dropout rate p

Original: BatchNorm, ReLU

Original: Conv16 × 4 × (( i + 1) // 2) ( k = 3 , d = 1) filters , stride =

Original: ( i + 1) // 2 , ReLU

Original: Add residual (apply point-wise conv first if c in ̸

Original: = c out )

Original: The output block consists of a BatchNorm layer, a ReLU activation, a linear layer, and a final activation layer which we modify according to the task learning objective, e.g., log softmax for classification and sigmoid for dense prediction.

Original: We set p = 0 in search and tune p as a hyperparameter for retraining.

Original: We use the WRN code provided here: https://github.com/meliketoy/wide-resnet.pytorch .

Original: 1D

Original: Tasks We use the 1D WRN [34] as the backbone for all 1D tasks.

Original: The model is made up ofresidual blocks with the following structure:

Original: Convc out ( k = 8 , d = 1) filters

Original: Dropout dropout rate p

Original: BatchNorm, ReLU

Original: Convc out ( k = 5 , d = 1) filters

Original: Dropout dropout rate p

Original: BatchNorm, ReLU

Original: Convc out ( k = 3 , d = 1) filters

Original: Dropout dropout rate p

Original: BatchNorm, ReLU

Original: In the original architecture, c out = 64 .

Original: We set c out to min(4 num_classes // 10+1 , 64) to account for simpler tasks with fewer class labels.

Original: The output block consists of a linear layer and a activation layer which we modify according to the task learning objective, e.g., log softmax for classification and sigmoid for dense prediction.

Original: We set p = 0 in search and tune p as a hyperparameter for retraining.

Original: We use the 1D WRN code provided here: https://github.com/okrasolar/pytorch-timeseries .

Original: A.5. D ASH Pipeline Hyperparameters Search

Original: • Epoch:• Optimizer: SGD(momentum=0.9, nesterov=

Original: True, weight_decay=5e-4) for both model weights and architecture parameters • Model weight learning rate: 0 . 1 for point prediction tasks, 0 .

Original: 01 for dense tasks

Original: [-• Architecture parameter learning rate: 0 . 
-]
Revised: [-• Architecture parameter learning rate: 0 . 
-]

Original: for point prediction tasks, 0 .

Original: 005 for dense tasks • Learning rate scheduling: decay by 0 .

Original: 2 at epoch• Gradient clipping threshold:• Softmax temperature:• Subsampling ratio: 0 .

Original: Hyperparameter tuning

Original: • Epoch:• Configuration space: – Learning rate: { 1 e- 1 , 1 e- 2 , 1 e- 3 } – Weight decay: { 5 e- 4 , 5 e- 6 } – Momentum: { 0 . 9 , 0 . 99 } – Dropout rate: { 0 , 0 . 05 }

Original: Retraining

Original: • Epoch:• Learning rate scheduling: for 2D tasks, decay

Original: [Table]

Original: A.6 Accuracy Results on NAS-Bench-360 with Error Bars

Original: [Table]

Original: PSICOV

Original: Cosmic

Original: [Table]

Original: -1 error(%) relative ℓ

Original: MAE1-

Original: [Table]

Original: [Table]

Original: A.8 Searched Architecture Visualization

Original: In this section, we give two example networks searched by D ASH to show that large kernel matters for diverse tasks.

Original: A.8.1 2D Example: Darcy Flow

Original: For this problem, D ASH generates a WRN 16-4 [33] for retraining.

Original: The network architecture consists of several residual blocks.

Original: For instance, we can use Block 64 , (7 , 1) , (9 , 3) to denote the residual block with the following structure:

Original: [Equation]

Original: where 64 is the output channel and BN denotes the BatchNorm layer.

Original: Note that size of a convolutional layer in the figure is proportional to the kernel size but not the number of channels.

Original: Then, an example network produced by D ASH for Darcy Flow looks like the following:

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: Block 256 , (3 , 7) , (7 , 7)

Original: BN+ReLU

Original: LinearChannel-Matching

Original: Since Darcy Flow is a dense prediction task, the last layer is a channel-matching (permutation+linear+permutation) layer instead of a pooling+linear layer for classification.

Original: A.8.2 1D Example: DeepSEA

Original: For this problem, D ASH generates a 1D WRN [34] for retraining.

Original: The network architecture consists of several residual blocks.

Original: For instance, we can use Block 64 , (3 , 1) , (5 , 3) , (7 , 5) to denote the residual block with the following structure:

Original: [Figure]

Original: where 64 is the output channel and BN denotes the BatchNorm layer.

Original: Then, an example network produced by D ASH for DeepSEA looks like the following:

Original: [Equation]

Original: [Figure]

Original: We can see that large kernels are indeed selected during search.

Original: A.9 Additional Results A.9. D ASH -TCN for NAS-Bench-

Original: D ASH works for all networks with a convolutional layer, so WRNs are not the only applicable backbone.

Original: Below, we provide the test errors of D ASH with the 1D Temporal Convolutional Network backbone on some 1D tasks:

Original: [Table]

Original: We did not include the results in the paper to simplify presentation.

Original: Also, using WRNs in our workflow allows us to provide a fully automated pipeline that generates decent-performing models as quickly (due to its small size) and easily (due to the code for training WRNs being easily found online) as possible for previously unexplored tasks.

Original: A.9. D ASH -ConvNeXt for ImageNet

Original: Though our motivation is not to compete in the crowded vision domain but to provide a general solution to less-studied domains, we show that D ASH is backward compatible with vision tasks by testing it on ImageNet-1K with two backbones of distinct scales.

Original: Our results show that D ASH generalizes to tasks with large input shape ( 3 × 224 × 224 ), dataset size (1.2M), and number of classes (1000).

Original: It improves the accuracy of the original models and searches efficiently regardless of the backbone used.

Original: We used Wide ResNet 16-4 (to be consistent with our workflow) and ConvNeXt-T [15] (a largescale CNN that has onpar performance with SoTA Transformers) as the backbones and performed experiments on 4 NVIDIA V100 GPUs.

Original: To demonstrate D ASH ’s efficiency, we first present the per-epoch search time (forward and backward time in secs) for three baselines over the search space

Original: K = { 3 , 5 , 7 , 9 , 11 } , D = { 1 , 3 , 7 } .

Original: A subset of 4096 images is used.

Original: [Table]

Original: We can see that D ASH ’s efficiency holds for both backbones.

Original: Though ConvNeXt has more parameters, it is searched faster than WRN as it has fewer conv layers and applies downsampling to the input.

Original: Then, we report D ASH ’s runtime vs. the train-time of the vanilla backbone (in hours).

Original: We let D ASH search for 10 epochs with subsampling ratio 0.2.

Original: (Re)training takes 50 and 100 epochs for WRN and

Original: ConvNeXt, respectively.

Original: [Table]

Original: Lastly, we report the top-1 accuracy of the searched vs. original models to show D ASH generalizes to large vision input.

Original: We trained ConvNeXt for 300 epochs.

Original: [Table]

Original: In general, D ASH improves backbone performance by adopting task-specific kernels.