Original: NAS-B ENCH -ASR: R EPRODUCIBLE N EURAL A RCHITECTURE S EARCH FOR S PEECH R ECOGNITION
Revised: NAS-B ENCH -ASR: R EPRODUCIBLE N EURAL A RCHITECTURE S EARCH FOR S PEECH R ECOGNITION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Powered by innovations in novel architecture design, noise tolerance techniques and increasing model capacity, Automatic Speech Recognition (ASR)  made giant strides in improving prediction accuracies over the past decade.
Revised: Powered by innovations in novel architecture design, noise tolerance techniques and increasing model capacity, Automatic Speech Recognition (ASR) has made giant strides in reducing word-error-rate over the past decade.

Original: ASR models are often trained with tens of thousand hours of high quality speech data to producethe state-of-the-art  results.
Revised: ASR models are often trained with tens of thousand hours of high quality speech data to produce state-of-the-art (SOTA) results.

Original: Industry-scale ASR model-training thus remainsas a computationally heavy and time-consuming procedure, and consequently  attracted little attentionthus far in adopting automatic techniques in exploring neural architecture variations.
Revised: Industry-scale ASR model training thus remains computationally heavy and time-consuming, and consequently has attracted little attention in adopting automatic techniques.

Original: Neural Architecture Search (NAS), on the other hand,  gained a lot of interest in the past few years for its ability in discovering state-ofthe-art architectures mainly in computer vision tasks.
Revised: On the other hand, Neural Architecture Search (NAS) has gained a lot of interest in the recent years thanks to its successes in discovering efﬁcient architectures, often outperforming handcrafted alternatives.

Original: However,  NAS approaches also suffer from the requirement of  a large-scale computing infrastructure to support training of a massive number of neural networks and are often difﬁcult to reproduce.
Revised: However, by changing the standard training process into a bi-level optimisation problem, NAS approaches often require signiﬁcantly more time and computational power compared to single-model training, and at the same time increase complexity of the overall process. As a result, NAS has been predominately applied to problems which do not require as extensive training as ASR, and even then reproducibility of NAS algorithms is often problematic.

Original: Lately, a number of attempts have been made to ameliorate the computational problem and improve the search turnaround time of NAS algorithms by introducing benchmark datasets like NAS-Bench-101, NAS-Bench-201, and NAS-NLP.
Revised: Lately, a number of benchmark datasets has been introduced to address reproducibility issues by providing NAS researchers with information about performance of different models obtained through exhaustive evaluation.

Original: These datasets, however, focus predominantly on computer vision and NLP tasks and thus suffer fromthe problem of limited coverage of application domains. 
Revised: However, these datasets focus mainly on computer vision and NLP tasks and thus suffer from limited coverage of application domains. In order to increase diversity in the existing NAS benchmarks, and at the same time provide systematic study of the effects of architectural choices for ASR, we release

Original: In this work we apply NAS for ﬁnding cell architecture for ASR modelsand release a comprehensive NAS-Bench dataset for reproducible NAS research.
Revised: NAS Bench -ASR – the ﬁrst NAS benchmark for ASR models

Original: The dataset consists of8 , 242 uniqueASR models trained on the TIMIT audio dataset, each starting from 3 seed initializations.
Revised: The dataset consists of , 242 unique models trained on the TIMIT audio dataset for three different target epochs

Original: Novelty of our dataset  includesconsideration for on-device deployability and inclusion of runtime measures of all the models ondiverse hardware platforms and settings. We further evaluate performances of a number of NAS algorithms on our dataset.
Revised: , and each starting from three different initializations The dataset also includes runtime measurements of all the models on a diverse set of hardware platforms.

Original: Finally, we show that cells in our search space for TIMIT transfer well to a much larger LibriSpeech dataset.
Revised: Lastly, we show that identiﬁed good cell struc tures in our search space for TIMIT transfer well to a much larger LibriSpeech dataset.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Innovations in Deep Neural Network (DNN) architecture design, data augmentation techniques and a continuous increase in the amount of available high quality training datasets, resulted in a massive reduction in ASR error rates over the past decade [1; 31; 39; 19; 33].
Revised: Innovations in Deep Neural Network (DNN) architecture design, data augmentation techniques and a continuous increase in the amount of available high quality training datasets, resulted in a massive reduction in ASR word-error-rate over the past decade [Amodei et al., 2016; Kim et al., 2019;

Revised: Park et al., 2019; Synnaeve et al., 2020].

Original: However, training ASR models to achieve state-of-the-art performanceaccuracy remains challenging as it requires computationally heavy training process, e.g., often thousands  GPU-hours are needed for good convergence [1; 17].
Revised: However, training ASR models to achieve state-of-the-art performance remains challenging as it requires computationally heavy training process, e.g., often thousands of GPU-hours are needed for good convergence [Amodei et al., 2016; Kahn et al., 2020].

Original: Furthermore, the requirement of hyper-parameter optimizations increases the computational loads in ASR training.
Revised: Furthermore, the requirement of hyper-parameter optimizations increases the computational loads in ASR training.

Original: Despite the system-level complexities in the training procedure, the importance of novel architecture design has proven extremely important in a variety of application domains including ASR [6; 34], computer vision [21; 14], and natural-language processing (NLP) [42; 7].
Revised: Despite the system-level complexities in the training procedure, the importance of novel architecture design has proven extremely important in a variety of application domains including ASR [Chiu & Raffel, 2018; Pratap et al., 2020], computer vision [He et al., 2016; Krizhevsky et al., 2012], and natural-language processing (NLP) [Devlin et al., 2019; Vaswani et al., 2017].

Original: However, architecture design is a non-trivial task and often depends on years of experience, domain knowledge of the researchers and is driven by empirical successes.
Revised: However, architecture design is a non-trivial task and often depends on years of experience, domain knowledge of the researchers and is driven by empirical successes.

Original: Over the past few years, the deep learning community has been witnessing a trend inincreased popularity of adopting automatic techniques for ﬁnding neural network architectures over  traditional hand-designed alternatives.
Revised: Over the past few years, the deep learning community is witnessing a trend in adopting automatic techniques to ﬁnd neural network architectures over more traditional hand-designed alternatives.

Original: NAS algorithms have proven themselves highly successful in discovering state-of-the-art architectures, in addition to ﬁnding the optimal set of parameters for them, in various computer vision tasks [3; 16; 23; 36; 40; 41].
Revised: NAS algorithms are highly successful in discovering state-of-the-art architectures in various computer vision tasks [Cai et al., 2020; Howard et al., 2019; Lee et al., 2020; Real et al., 2018; Tan et al., 2019; Tan & Le, 2019].

Original: However, many of them suffer from high computational requirements, necessitating a large number of architecture variations to be trained [46].
Revised: However, many of them suffer from high computational demands, requiring a large number of architecture variations to be trained [Zoph & Le, 2017].

Original: Furthermore, NAS algorithms are often difﬁcult to reproduce as they use diverse training procedures and employ different architecture search spaces [26; 37].
Revised: Furthermore, NAS algorithms are often difﬁcult to reproduce by different researchers, mainly due to a non-standard use of training settings, e.g., hyperparameters, and subtle variations in the architecture search spaces [Li &

Revised: Talwalkar, 2019; Sciuto et al., 2020].

Original: Recently,  attempts have been made to mitigate these problems by releasing various benchmark datasets forNAS to the  community [44; 9; 38; 20].
Revised: Recently, a number of attempts have been made to mitigate these problems by releasing various benchmark datasets for the NAS research community [Dong &

Revised: Yang, 2020; Klyuchnikov et al., 2020; Siems et al., 2020; Ying et al., 2019].

Original: These datasets usually provide a  mapping between an architecture variant and their post training performances, which can be used efﬁciently by a NAS algorithm, i.e., the time-consuming training tasks can be replaced by fast dataset queries while searching for better architectures.
Revised: These datasets usually provide a direct mapping between an architecture variant and its post training performances, which can be used efﬁciently by a NAS algorithm speeding up the search process and, at the same time, providing common, fully reproducible environment for assessment and comparison of different algorithms.

Original: Initial attempts of creating benchmark datasets predominantly focus on image classiﬁcation tasks (with only one  targeting NLP tasks), and thus suffer from poor application coverage.
Revised: Initial attempts of creating benchmark datasets predominantly focus on image classiﬁcation tasks (with only one existing work targeting NLP at the time of this writing), and thus suffer from poor application coverage.

Original: In this work we address the lack of coverage problem by introducing a  NAS-benchmak dataset in the domain of ASR, to our best knowledge the very ﬁrst of its kind.
Revised: We address the lack of coverage problem by introducing a new NAS-benchmak dataset in the domain of ASR, to our best knowledge the very ﬁrst of its kind.

Original: To build the dataset, we have trained 8 , 242 unique convolutional neural network architectures on the TIMIT dataset [12].
Revised: To build the dataset, we have trained 8 ,unique convolutional neural network architectures on the TIMIT dataset [Garofolo et al., 1993].

Original: We consider convolutional architectures due to their recent success in beating state-of-the-art models in  ASR [34; 13].
Revised: We consider convolutional architectures due to their recent successes in the domain of ASR [Pratap et al., 2020; Hannun et al., 2019].

Original: Moreover, convolution based architectures are computationally efﬁcient to run on mobile CPUs, thus favouring real-time on-device deployment.
Revised: Moreover, convolution based architectures are computationally efﬁcient to run on mobile devices, thus favouring real-time on-device deployment.

Original: Our dataset contains multiple runs of the entire training procedure, spanning 3 different initializations of the network parameters.
Revised: Our dataset contains multiple runs of the entire training procedure of an architecture, spanning three initializations of the network parameters and three target epochs, amounting a total of 74 , 178 = 8 , 242 × 3 ×training runs.

Original: In addition to the per epoch validation and ﬁnal test metrics, such as Phoneme Error Rate (PER), and CTC loss, we also provide run-times of individual architecturesin varying batch sizes ondifferent hardware platforms including desktop and embedded GPUs.
Revised: In addition to the per epoch validation and ﬁnal test metrics, such as Phoneme Error Rate (PER), and CTC loss, we also provide run-times of the architectures on desktop and embedded GPUs for varying batch size.

Original: We consider a number of NAS algorithms [46; 36; 10; 25; 26] and show their performance on our search space, showcasing potential challenges and differences from their performance onthe existing benchmarks.
Revised: Furthermore, we compare a number of NAS algorithms [Zoph & Le, 2017; Real et al., 2018; Dudziak et al., 2020; Li et al., 2017; Li & Talwalkar, 2019] on our search space, highlighting potential challenges and differences compared to their performances on existing NAS benchmark dataset.

Original: Lastly, we show the transferability of the top architecture cells found on TIMIT to a much larger Librispeech [30] dataset.
Revised: Lastly, we show the transferability of the top architecture cells found on TIMIT to a much larger Librispeech dataset [Panayotov et al., 2015].

Original: In summary, the contributions of this paper are:
Revised: In summary, the contributions of this paper are:

Original: • Design of ASR NAS Search Space.
Revised: • Design of ASR NAS Search Space.

Original: ASR NAS-Bench is a ﬁrst-of-its-kind search space for convolutional speech models.
Revised: ASR NAS-Bench is a ﬁrst-of-its-kind search space for convolutional speech models.

Original: It facilitates the reproducible study of ASR through NAS methods and thus ﬁlls a important gap in  literature.
Revised: It facilitates the reproducible study of ASR through NAS methods and thus ﬁlls an important gap in the literature.

Original: The associated(soon to be publicly released) dataset consists of 8 , 242 unique cells and contains validation and test metrics along with model parameters, FLOPs and on-device run-times.
Revised: The associated dataset consists of 8 , 242 unique cells and contains validation and test metrics along with model parameters, FLOPs and on-device run-times 1 .

Original: • Enabling NAS for Large-scale ASR.
Revised: • Enabling NAS for Large-scale ASR.

Original: Prohibitive training times for non-toy ASR datasets, has prevented NAS from strongly inﬂuencing the evolution of ASR architecture design.
Revised: Prohibitive training times for non-toy ASR datasets, has prevented NAS from strongly inﬂuencing the evolution of ASR architecture design.

Original: We show that ASR NAS-Bench is able to support the discovery of cell structures that generalize even to largescale datasets like Librispeech – a key breakthrough .
Revised: We show that ASR NAS-Bench is able to support the discovery of cell structures that generalize even to largescale datasets like Librispeech – a key breakthrough .

Original: We believe the methodological decisions in this paper will act as a blueprint for future work where NAS plays a prominent role in ASR design.
Revised: We believe the methodological decisions in this paper will act as a blueprint for future work, where NAS plays a prominent role in ASR design.

Original: • Validating Existing NAS Algorithm Design.
Revised: • Validating Existing NAS Algorithm Design.

Original: Existing understanding of NAS is grossly inﬂuenced by image-based tasks.
Revised: Existing understanding of NAS is grossly inﬂuenced by image-based tasks.

Original: By systematically benchmarking popular NAS algorithms, under a rich ASR search space, our ﬁndings provide otherwise lacking scientiﬁc support for prior results.
Revised: By systematically benchmarking popular NAS algorithms, under a rich ASR search space, our ﬁndings provide otherwise lacking scientiﬁc support for prior results.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: NAS benchmarks.
Revised: NAS Benchmarks.

Original: Ying et al.
Revised: Ying et al.

Original:  inroduced NAS-Bench-101 dataset in an attempt to address the difﬁculties in reproducing NAS research [44].
Revised: [2019] inroduced NAS-Bench-101 dataset in an attempt to address the difﬁculties in reproducing NAS research.

Original: It containsdata about over 400 K unique image classiﬁcation models trained on the CIFAR10 dataset.
Revised: It contains over 400 K unique image classiﬁcation models trained on the CIFAR10 dataset.

Original: Despite being the biggest NAS dataset yet, not all NAS algorithms can utilize the dataset due to the restrictions it imposes on the maximum number of edges to limit the search space size.
Revised: Despite being the biggest NAS dataset yet, not all NAS algorithms can utilize the dataset due to the restrictions it imposes on the maximum number of edges to limit the search space size.

Original:  NAS-Bench-201 [9], which consists of 15 Kmodels, was designed similarly to its precursor and thus also focuses on image classiﬁcation models, but tries to overcome the limitations of the 101 dataset and includes more diagnostic data.{+data [Dong & Yang, 2020]. Concurrently to NAS-Bench-201, NAS-Bench-1shot
 [Zela et al., 2020] and NAS-Bench
+}
Revised: To mitigate the limitations, NAS-Bench-201 was introduced, which contains 15 K image classiﬁcation models and includes more diagnostic {+data [Dong & Yang, 2020]. Concurrently to NAS-Bench-201, NAS-Bench-1shot
 [Zela et al., 2020] and NAS-Bench
+}

Original: Concurrently to
Revised: [Siems et al., 2020] were introduced.

Original: NAS-Bench-201, NAS-Bench-1shot1 was published with focus on benchmarking NAS which use weight sharing [45]. NAS-Bench-301[38], points out the need for surrogate functions to scale NAS benchmarking. The authors prepared a benchmark based on the DARTS search space [27] with approximately 10 18 models.
Revised: NAS-Bench-1shot1 focuses on benchmarking NAS with weight sharing, whereas, NAS-Bench-301 points out the need for surrogate functions for scaling and uses the DARTS search space [Liu et al., 2019] with approximately 10 18 models.

Original: Finally, NAS-Bench-NLP is a NAS benchmark containing models constructed by considering custom recurrent cells, which are used to replace traditional layers like 
Revised: Lastly, NASBench-NLP [Klyuchnikov et al., 2020] contains models with custom recurrent cells, which are used to replace traditional layers like LSTMs.

Original: LSTMs, totaling to 14 K different architectures[20] trained on  language modelling task.
Revised: The dataset contains around 14 K different architectures trained on a language modeling task.

Original: To the best of our knowledge, it is the only NAS benchmark that goes beyond the task of image classiﬁcation.

Original: NAS in the speech domain.
Revised: NAS in Audio Modeling.

Original: Very recently, there has been a visible increase in interest in  NAS for speech-related tasks, such as keyword spotting [29; 28], speaker veriﬁcation [8; 35], or acoustic scene classiﬁcation [24].
Revised: Recently, there has been an increasing interest in applying NAS in speechrelated tasks, such as keyword spotting [Mo et al., 2020; Mazzawi et al., 2019], speaker veriﬁcation [Ding et al., 2020; Qu et al., 2020], and acoustic scene classiﬁcation [Li et al., 2020].

Original: Consequently, NAS has also appeared in some works concerning speech recognition [15; 5; 18; 2].
Revised: NAS has also been applied in ASR [He et al., 2020; Chen et al., 2020; Kim et al., 2020; Baruwa et al., 2019].

Original:  Chen et al.
Revised: For example, Chen et al.

Original:  used a search methodology based on the vanilla DARTS to op- timize a CNN-based feature extractor which is then followed by a ﬁxed Bi-LSTM module and multiple output heads [5].
Revised: [2020] used a search methodology based on the vanilla DARTS to optimize a CNN-based feature extractor, followed by a ﬁxed Bi-LSTM module and multiple output heads.

Original: They focused their evaluation on performance of the NAS-discovered models in mono- and multilingual settings using the Full Language Pack from IARPA BABEL [11] showing improvements over a VGG-based extractor.
Revised: The authors evaluated the NAS-discovered models under mono- and multi-lingual settings, using the Full Language Pack from IARPA BABEL [Gales et al., 2014], showing improvements over a VGG-based extractor.

Original: Unlike the above, Kim et al.  considered evolution-based search to optimize a micro-cell used in a transformer-based architecture [18].
Revised: In contrast, Kim et al. [2020] considered evolution-based search to optimize a micro-cell used within a transformer architecture.

Original: The evaluation was done usinga monolingual setting only but with consideration of both English and Korean-based datasets of approximately 100 hours of speech each.
Revised: The evaluation was done using English and Korean dataset of approximately 100 hours of speech each, under monolingual setting.

Original: Similar to Chen et al. He et al.
Revised: Similarly, He et al.

Original:  used differentiable search (although using P-DARTS [4] as their base) to optimize a convolution-based modelbut without the recurrent ”tail” [15].
Revised: [2020] used differentiable search, using P-DARTS [Chen et al., 2019] as the base, to optimize a convolutional model without the recurrent tail .

Original: Also, unlike the previous ones, their evaluation only compromises single language (Chinese Mandarin) but using various  datasets spanning between 170 and 10k hours of speech.
Revised: Unlike the previous approaches, the evaluation is done on Mandarin, while using various training datasets spanning between 170 and 10 K hours of speech.

Original: The most similar to ours is the work by Baruwa et al.
Revised: Closest to our work is the work by Baruwa et al.

Original: They also considerperformance on both TIMIT and LibriSpeech datasets, butunlike us they studied them independently rather than using TIMIT as a proxy for LibriSpeech.
Revised: [2019], where the authors consider both TIMIT and LibriSpeech datasets, but studied them independently rather than using TIMIT as a proxy for LibriSpeech.

Original: Further differences involve search space design – the authors only focused on selecting operations which were then placed in a ﬁxed, feed-forward manner (although the paper lacks some details so it is hard to say anything for sure), andalso allowed to interleave convolutions with recurrent blocks.
Revised: Further differences include variations in search space design, e.g., the authors only considered operations arranged in a ﬁxed feed-forward manner, and allowed interleaving convolutions with recurrent blocks.

Original: Lastly,we would like to emphasize that all the current work on NAS for ASR focuses on using the searchsolely to improve results – they provide important empirical proofs that NAS can be successfully used in relevant domains but at the same time they  lack solid foundations needed for analysis and reasoning about the overall process, its limitations etc.
Revised: Lastly, current work on NAS for ASR focus mainly on using the search to improve prediction accuracy, however, they often lack solid foundations needed for analysing and reasoning about the overall search process and identifying its limitations.

Original: This gap in the existing literature is exactly what we try to ﬁll by providing information about large-scale study of the effects of architectural changes on ASR models.
Revised: This work presents a large-scale study on the effects of architectural changes on ASR models.

Original:  ASR NAS-B ENCH
Revised:  ASR NAS-B ENCH

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure] Overall, the cell architecture space has unique The main purpose of the ASR NAS-Bench dataset is to provide a direct mapping from an architecture instance in themicro-architecture search-space (§ 3.1) to its training time and ﬁnal performance metrics.
Revised: The main purpose of the ASR NAS-Bench dataset is to provide a direct mapping from an architecture instance in the search-space (§3.1) to its training time and ﬁnal performance metrics.

Original: The mapping is designed for any NAS algorithm to quickly navigate the architecture space without incurring the time-consuming and computationallyheavy training procedure.
Revised: The mapping is designed for any NAS algorithm to quickly navigate the architecture space without requiring the time-consuming and computationally-heavy training procedure.

Original: For micro-architecture search we use the TIMIT dataset [12] (§ 3.2) and present details of the pilot experimentconducted to select  hyperparameters(§ 3.3), used to train 8 ,models (§ 3.4).
Revised: For architecture search we use the TIMIT dataset [Garofolo et al., 1993] and conduct a pilot experiment to select suitable hyperparameters used to train 8 , 242 unique models.

Original: 3. ASR A RCHITECTURE S EARCH S PACE
Revised: 3. ASR A RCHITECTURE S EARCH S PACE

Original: In line with existing NAS approaches [27; 32] and 
Revised: In-line with existing work [Liu et al., 2019; Pham et al., 2018] and NAS Benchmarks

Original: NAS Benchmarks [44; 9], we restrict our search to a small feed-forward neural network topologyspace, commonly known as cells .
Revised: [Ying et al., 2019; Dong & Yang, 2020], we restrict our search to a small feed-forward neural network topologyspace, commonly known as cells .

Original: We repeat and arrange a chosen cell to construct a predeﬁned macroarchitecture, which is then trained on the TIMIT dataset.
Revised: We repeat and arrange a chosen cell to construct a predeﬁned macro-architecture, which is then trained on the TIMIT dataset.

Original: Micro-Architecture.
Revised: Micro-Architecture.

Original: A micro-architecture or a cell , as shown in Figure 1(a), is represented by a directed acyclic graph (DAG).
Revised: A micro-architecture or a cell , as shown in Figure 1(a), is represented by a directed acyclic graph (DAG).

Original: We consider DAGs with four nodes representing tensors t 1 , . ..
Revised: We consider DAGs with four nodes T 1 , . . , T 4 with corresponding incident tensors t 1 , . .

Original: , t 4 and allow two types of edges: main and skip connection edges.
Revised: , t 4 and allow two types of edges: main and skip connection edges.

Original: A main edge connects two successive nodes t i − 1 and t i in the graph as shown by the solid line-arrows in Figure 1(a).
Revised: A main edge connects two successive nodes T i − 1 and T i in the graph as shown by the solid line-arrows in Figure 1(a).

Original: A skip connection edge on the other hand, can connect any two nodes  j and  i , with the constraint j < i and are depicted as dotted line-arrows in Figure 1(a).
Revised: A skip connection edge on the other hand, can connect any two nodes T j and T i , with the constraint j < i and are depicted as dotted line-arrows in Figure 1(a).

Original: Each edge e j → i represents an operation on the value of node t j .
Revised: Each edge e j → i represents an operation on the tensor t j .

Original: The value of a node t i is computed by summing the results of operations done by all incident edges on t i (i.e., e j → i , where j < i ). 
Revised: The tensor t i is computed by summing the results of operations done by all incoming edges on T i (i.e., e j → i , where j < i ). [Figure]

Original: We consider a choice of six operations for the main edges: linear operation, four convolution operations distinguished by choices of ( kernel size , dilation ) ∈
Revised: We consider a choice of six operations for the main edges: linear operation, four convolution operations distinguished by choices of ( kernel size , dilation ) ∈

Original: {( 5 , 1 ) , ( 5 , 2 ) , ( 7 , 1 ) , ( 7 , 2 )} , and a zero operation, which outputs a tensor of zeros with the same shape and type as its input.
Revised: {( 5 , 1 ) , ( 5 , 2 ) , ( 7 , 1 ) , ( 7 , 2 )} , and a zero operation, which outputs a tensor of zeros with the same shape and type as its input.

Original: Skip con- nection operations on the other hand, can be either the identity operation or the zero operation.
Revised: Skip connection operations on the other hand, can be either the identity operation or the zero operation.

Original: We used L2 kernel regularizer with convolution operations and dropouts with linear operations.
Revised: We used L2 kernel regularizer with convolution operations and dropouts with linear operations.

Original: Within the convolution operations, the size of the kernels (e.g., 5 or 7 ) is chosen as a trade-off between the audio context duration and the model size.
Revised: Within the convolution operations, the size of the kernels (e.g., 5 or 7 ) is chosen as a trade-off between the audio context duration and the model size.

Original: Similarly two dilation factors are considered (e.g., 1 or 2 ) to investigate the trade-off between audio context duration and time resolution.
Revised: Similarly two dilation factors are considered (e.g., 1 or 2 ) to investigate the trade-off between audio context duration and time resolution.

Original: As the ﬁnal step of a cell, the value of t 4 is further passed through a layer normalization to produce the output of the cell.
Revised: As the ﬁnal step of a cell, the value of t 4 is further passed through a layer normalization to produce the output of the cell.

Original: These design choices are made considering the search space that can be used by the vast majority of the NAS algorithms.
Revised: These design choices are made after we have considered the search space that can be used by the vast majority of the NAS algorithms.

Original: Macro-Architecture.
Revised: Macro-Architecture.

Original: The macro architecture considered in this paper is  illustrated in Figure 1(b), which follows a sequential computationand is composed of four blocks followed by a unidirectional LSTM1 and a linear layer.
Revised: One of our focuses in this benchmark is on-device deployment, so the macroarchitecture is made up of convolution and unidirectional LSTM blocks as they are computationally efﬁcient to run on mobile CPUs. The macro architecture, illustrated in Figure 1(b), is a sequential computation composed of four blocks followed by a unidirectional LSTM and a linear layer.

Original: Individual blocks are composed of a convolution layer, an activation layer (ReLU), a layer normalization and a composition of C i ( i = 1 , 2 , 3 , 4 ) search cells, with the same micro architecture across all the blocks.
Revised: Individual blocks are composed of a convolution layer, an activation layer (ReLU), a layer normalization and a composition of C i ( i = 1 , 2 , 3 , 4 ) search cells, with the same micro architecture across all the blocks.

Original: Each block is parametrized by three parameters F i , K i and S i .
Revised: Each block is parametrized by three parameters F i , K i and S i .

Original: These are used to deﬁne the number of ﬁlters F i , the kernel size K i and the stride S i of the convolution layer.
Revised: These are used to deﬁne the number of ﬁlters F i , the kernel size K i and the stride S i of the convolution layer.

Original: Note that F i is also the number of ﬁlters of the convolution layers inside the search cell.
Revised: Note that F i is also the number of ﬁlters of the convolution layers inside the search cell.

Original: We use the following set of parameters to deﬁne the macro-architecture while performing all micro-architecture training on the TIMIT dataset: C 1 ∶ 4 = [ 3 , 4 , 5 , 6 ] , F 1 ∶ 4 =
Revised: We use the following set of parameters to deﬁne the macro-architecture while performing all micro-architecture training on the TIMIT dataset: C 1 ∶ 4 = [ 3 , 4 , 5 , 6 ] , F 1 ∶ 4 =

Original: [ 600 , 800 , 1000 , 1200 ] , K 1 ∶ 4 =
Revised: [ 600 , 800 , 1000 , 1200 ] , K 1 ∶ 4 =

Original: [ 8 , 8 , 8 , 8 ] and[-S 
-]
Revised: [ 8 , 8 , 8 , 8 ] and[-S 
-]

Original:  ∶ 4 =
Revised: S 1 ∶ 4 =

Original: 3. TIMIT D ATASET
Revised: 3. TIMIT D ATASET

Original: TIMIT [12] is one of the earliest datasets designed for evaluating and benchmarking phoneme ASR systems.
Revised: TIMIT [Garofolo et al., 1993] is one of the earliest datasets designed for evaluating and benchmarking phoneme ASR systems.

Original: It is ideally suited for neural architecture search experiments due to its small size and high quality transcriptions.
Revised: It is ideally suited for neural architecture search experiments due to its small size and high quality transcriptions.

Original: It comprises of 6 , 300 utterances from 630 speakers, amounting to 5 .hours of speech.
Revised: It comprises of 6 , 300 utterances from 630 speakers, amounting to 5 . 4 hours of speech.

Original: We use the standard training partition of 3 , 696 utterances from 462 speakers.
Revised: We use the standard training partition of 3 , 696 utterances from 462 speakers.

Original: Following [22], we split the core test dataset into a test partition, consisting of 24 speakers, and a validation partition.
Revised: Following Lee & Hon [1989], we split the core test dataset into a test partition, consisting of 24 speakers, and a validation partition.

Original: Following kaldi Timit-s5 recipe, the original 61 phonemes are mapped into a set of 48 phonemes, which forms the output layer targets for all the models.
Revised: Inline with kaldi Timit-s5 recipe, the originalphonemes are mapped into a set of 48 phonemes, which forms the output layer targets for all the models.

Original: These 48 phonemes were further folded to a set of 39 during evaluations [22].
Revised: These 48 phonemes were further folded to a set of 39 during evaluations [Lee & Hon, 1989].

Original: We use 80-dimensional log-mel spectrograms computed over 25 ms sliding window with a stride of 10 ms as input features.
Revised: We use 80-dimensional log-mel spectrograms computed over 25 ms sliding window with a stride of 10 ms as input features.

Original: During training, we employ a curriculum learning strategy [1], where only audio utterances shorter than 1s are presented in the ﬁrst 2 epochs, followed by audio utterances shorter than 2s for the next two epochs, and then the complete training set for the rest of the epochs.
Revised: During training, we employ a curriculum learning strategy [Amodei et al., 2016], where we begin training by iterating twice over audio utterances shorter than 1s, then twice over audio utterances shorter than 2s as a warmup phase followed by training on the entire dataset.

Original: For efﬁciency, we also use a batch bucketing strategy, where a batch size of 64 is used for audio utterances smaller than 2 s, and a batch size of 32 is used otherwise.
Revised: For efﬁciency, we also use a batch bucketing strategy, where a batch size of 64 is used for audio utterances smaller than 2 s, and a batch size of 32 is used otherwise.

Original: We used CTC beam-search decoder with beam-size of 4.
Revised: We used CTC beam-search decoder with beam-size of 12.

Original: We did not use any language model in our experiments in order to avoid its confounding impact and its HPO (i.e., weighting factors).
Revised: Note, we did not use any language model in our experiments in order to avoid further HPO (e.g., weighting factors) optimization.

Original: This also helped us to keep the architecture search for Acoustic Model (AM) tractable.
Revised: This also helped us to keep the architecture search for Acoustic Model (AM) tractable.

Original: All training experiments on TIMIT report the PER on validation and test partitions.
Revised: All training experiments on TIMIT report the PER on validation and test partitions.

Original: 3. P ILOT E XPERIMENT
Revised: 3. P ILOT E XPERIMENT

Original: As different training procedures can potentially lead to substantially different results [44], in this work we employ a single general training procedure for all models.
Revised: As different training procedures can potentially lead to substantially different results [Ying et al., 2019], in this work we employ a single general training procedure for all models.

Original: Moreover, we use the same parameter set, e.g., F, K, S , learning rate, and decay across all 8 , 242 models.
Revised: Moreover, we use the same parameter set, e.g., F, K, S , learning rate, and decay across all 8 , 242 models.

Original: In order to select good values of the parameters, we conducted a range of training experiments, where we performed  grid-search to ﬁnd good macro structure parameters and optimizer settings.
Revised: In order to select good values of the parameters, we conducted a range of training experiments, where we performed a grid-search to ﬁnd good macro structure parameters and optimizer settings.

Original: Selecting macro structure parameters.
Revised: Macro Structure Parameter Selection.

Original: Since our macro structure  has four blocks, we searched over tuples of size-four for each of the macro structure parameters ( F, K, S ). Speciﬁcally, we considered number of ﬁlters F ∈ {[ 600 , 800 , 1000 , 1200 ] , [ 900 , 1100 , 1300 , 1500 ] , [ 1200 , 1300 , 1500 , 1700 ]} , kernel sizes
Revised: The macro structure considered in this work has four main blocks (see Figure 1(a)) and we considered the following variations in the pilot study: ﬁlters F ∈ {[ 600 , 800 , 1000 , 1200 ] , [ 900 , 1100 , 1300 , 1500 ] , [ 1200 , 1300 , 1500 , 1700 ]} , kernel sizes

Original: K ∈ {[ 6 , 8 , 10 , 12 ] , [ 8 , 8 , 8 , 8 ] , [ 10 , 10 , 10 , 10 ] , [ 12 , 12 , 12 , 12 ]}  and time reduction via strides
Revised: K ∈ {[ 6 , 8 , 10 , 12 ] , [ 8 , 8 , 8 , 8 ] , [ 10 , 10 , 10 , 10 ] , [ 12 , 12 , 12 , 12 ]} , and time reduction via strides

Original: S ∈ {[ 2 , 2 , 2 , 1 ] , [ 1 , 1 , 2 , 2 ] , [ 2 , 2 , 1 , 1 ]} .
Revised: S ∈ {[ 2 , 2 , 2 , 1 ] , [ 1 , 1 , 2 , 2 ] , [ 2 , 2 , 1 , 1 ]} .

Original: Selecting optimizer settings for training.
Revised: Optimizer Setting for Training.

Original: Next, we also explored good ranges of the learning rate, decay factor and start epoch for an exponential decay learning rate scheduler used in conjunction with the Adam optimizer.
Revised: We further explored good ranges of the learning rate, decay factor, and start epoch for an exponential decay learning rate scheduler used in conjunction with the Adam optimizer.

Original: Speciﬁcally, we considered the learning rate (LR) in [ 10 − 3 , 10 − 4 , 5 ∗ 10 − 5 , 10 − 5 ] , the decay factor in [ 0 . 9 , 0 .
Revised: Speciﬁcally, we considered the learning rate (LR) ∈ [ 10 − 3 , 10 − 4 , 5 ∗ 10 − 5 , 10 − 5 ] , the decay factor ∈ [ 0 . 9 , 0 .

Original: 95 , 0 . 99 ] , and the LR decay start epoch in [ 5 , 10 , 15 ] .
Revised: 95 , 0 . 99 ] , and the LR decay start epoch ∈ [ 5 , 10 , 15 ] .

Original: Finally, we randomly chose 5 micro architectures among all possible 8 , 242 cells and for each cell we conducted the grid search over the aforementioned parameters.
Revised: Finally, we randomly chose ﬁve micro architectures among all possible 8 , 242 cells and for each cell we conducted the grid search over the aforementioned parameters.

Original: Speciﬁcally, for each cell we created a model from a particular macro architecture which we  trained via a particular optimizer setting on TIMIT.
Revised: Speciﬁcally, for each cell we created a model from a particular macro architecture, which we then trained by selecting a particular optimizer setting on TIMIT.

Original: Wethen identiﬁed the parametric setting that resulted in the lowest PER across most 5 cells in these experiments and used the same parameters in all latter model training experiments.
Revised: We identiﬁed the parametric setting that resulted in the lowest PER across all ﬁve cells in the experiments and used the same parameters in all latter model training experiments.

Original: 3. T RAINING F RAMEWORK FOR NAS-B ENCH -ASR{+The best macro structure parameters are presented above (see §3.1), whereas the best LR was 10 − 4 , and the decay factor and start epoch were: ( i ) 0 . 9 and 5 for target epoch 40 , ( ii ) 0 .and 
 for target epoch 10 , and ( iii ) 0 .+}
Revised: {+The best macro structure parameters are presented above (see §3.1), whereas the best LR was 10 − 4 , and the decay factor and start epoch were: ( i ) 0 . 9 and 5 for target epoch 40 , ( ii ) 0 .and 
 for target epoch 10 , and ( iii ) 0 .+}

Revised: 398 and 1 for target epoch 5 respectively.

Revised: The decay factor and start epoch are chosen such that the learning rate of the optimizer reaches the same value at the end of the target epoch of training.

Revised: 3. T RAINING S ETUP

Original: Individual models are trained using a TensorFlow-based training pipeline running on a single GPU.
Revised: Individual models are trained using a TensorFlow-based training pipeline running on a single GPU.

Original: We leveraged NVIDIA V100 and P40 GPUs, and decreased training time by increasing throughput via the bucketing strategy based on the audio length.
Revised: We leveraged NVIDIA V100 and P40 GPUs, and decreased training time by increasing throughput via the bucketing strategy based on the audio length.

Original: Metrics such as CTC loss and Phoneme Error
Revised: Metrics such as CTC loss and Phoneme Error

Original: Rate (PER) for TIMIT were used to measure model performance.
Revised: Rate (PER) for TIMIT were used to measure model performances.

Original: Speciﬁcally, train and validation metrics were logged at each epoch, and the test metrics were logged at the end of training for the best model (refered to as ﬁnal test PER ).
Revised: Speciﬁcally, train and validation metrics were logged at each epoch, and the test metrics were logged at the end of training for the best model (refered to as ﬁnal test PER ).

Original: Our dataset contains logs of each of the 8 , 242 models trained with three different seedsand for 40 epochs.
Revised: Our dataset contains logs of each of the 8 , 242 models trained with three different seeds and for three target epochs (5, 10 and 40), thus generating a total of 74 , 178 model training traces.

Original: Additionally, we computed the number of parameters and ﬂoating point operations (FLOPs) for each of the architectures and measured their latency on two commonly used hardware platforms.
Revised: Additionally, we computed the number of parameters and ﬂoating point operations (FLOPs) for each of the architectures and measured their latency on two commonly used hardware platforms: Tesla 1080Ti and Jetson Nano.

Original:  A NALYSIS OF NAS-B ENCH -ASR D ATASET
Revised:  A NALYSIS OF NAS-B ENCH -ASR D ATASET

Original: As described in §3.1, our cell search space is constructed by selecting 3 main operations from a set of 6 candidates, and then selecting either identity or zeroize operation for the 6 skip connection edges (c.f., Figure 1(a)).
Revised: As described in §3.1, our cell search space is constructed by selecting 3 main operations from a set of 6 candidates, and then selecting either identity or zeroize operation for the 6 skip connection edges (see Figure 1(a)).

Original: This amounts to 13 , 824 ( 6 3 × 2 6 ) possible instances for the search-space. 
Revised: This amounts to 13 , 824 ( 6 3 × 2 6 ) possible instances for the search-space. How-

Original: However, the use of zero operations introduces computational graph-isomorphism .
Revised: [Table] [Table] ever, the use of zero operations introduces computational graph-isomorphism .

Original: To identify unique architectures, following [44; 10], we ﬁrst identify nodes with zero operations and disconnect them from their neighbors.
Revised: To identify unique architectures, following [Ying et al., 2019; Dudziak et al., 2020], we ﬁrst identify nodes with zero operations and disconnect them from their neighbors.

Original: Then we perform a reachability test and remove all nodes that can not be reached from the input or output nodes of the cell.
Revised: Then we perform a reachability test and remove all nodes that can not be reached from the input or output nodes of the cell.

Original: After this we end up with a minimized graph representation, which we hash using an iterative graph invariant approach [43].
Revised: Finally, we end up with a minimized graph representation, which we hash using an iterative graph invariant approach [Ying, 2019].

Original: After accounting for isomorphism and discarding the empty graph from the list of valid models, we found 8 , 242 unique architectures, out of which 8 , 000 are without any zero operations.
Revised: After accounting for isomorphism and discarding the empty graph from the list of valid models, we found 8 , 242 unique architectures, out of which 8 , 000 are without any zero operations.

Original: Distribution of Model Performances.
Revised: Distribution of Model Performances.

Original: The ﬁrst observation from analyzing the model training results is that the majority of the models  converged and achieved a ﬁnal test PER within a small range.
Revised: Analyzing the model training results we found that the majority of the models have converged and achieved a ﬁnal test PER within a small range.

Original: Speciﬁcally, 82.86% (6,829) of the models fall below a PER of 26%, with the best model reaching  PER of 21.05%.
Revised: Speciﬁcally, 83 . 3 % ( 6 , 869 ) of the models fall below a PER of 26 %, with the best model reaching test PER of 20 . 83 %.

Original: On the other hand, the remaining 17.14% of the models form a rather evenly-distributed tail, achieving PER values between 26% to 100%.
Revised: The remaining 16 . 7 % of the models form a rather evenly-distributed tail, achieving PER values between 26% to 100%.

Original: Because of the high density of points below 26% PER and overall high range of PER values, we decided to conduct our analysis by considering three subgroups of models: T OP models are the top 1000 architectures according to test PER, M IDDLE are models at positions between 2000 and 3000 in the test PER ranking, and B OTTOM are between 7000 and 8000.
Revised: Because of the high density of points below 26% PER and overall high range of PER values, we conducted our analysis by considering three subgroups of models: T OP models are the top 1 , 000 architectures according to test PER, M IDDLE are models at positions between 2 , 000 and 3 , 000 in the test PER ranking, and B OTTOM are between 7 , 000 and 8000 .

Original: Correlation between Validation and Test PERs.
Revised: Correlation Between Validation and Test PERs.

Original: In Figure 2 we show the correlation between theﬁnal test PER and the best validation PER.
Revised: Figure 2 shows the correlation between the test PER and the best validation PER.

Original: We use best validation PER because the ﬁnal test PER was computed on the checkpoint which had the best validation score over all epochs.

Original: The results show there is  moderate correlation for top models, weak for middle ones, and strong for worst 
Revised: The results indicate that there is a moderate correlation for top models, weak for middle ones, and strong for worst ones.

Original: [Table] [Table] ones. High-level results are summarized numerically in Table 1.
Revised: An overall summary is also presented in Table 1.

Original: In general, correlation between validation and test accuracy among the top performing models is signiﬁcantly lower than for image classiﬁcation benchmarks – which isadditionally highlighted inTable 2.
Revised: In general, correlation between validation and test accuracy among the top performing models is signiﬁcantly lower than for image classiﬁcation benchmarks, which is highlighted in

Original: We suspect that this fact might pose additional challenge to NAS algorithms.
Revised: Table 2, can potentially pose additional challenge to NAS algorithms.

Original: Distribution of Operations.
Revised: Distribution of Operations.

Original: Figure 3 (ﬁrst row) presents the distribution of selected operations among main edges across top, middle and bottom most models in terms of performances.
Revised: Figure 3 (ﬁrst row) presents the distribution of selected operations among main edges across top, middle and bottom most models.

Original: We see top models have the highest count for linear operation, while the middle and bottom groups havethe lowselection of linear operation but higherfor convolution operation.
Revised: We found that the top models have the highest number of linear operations on the main edges, while the middle and bottom groups have low linear operations but higher convolution operations.

Original: At the same time, as shown in Figure 3 (second row), the worst models (bottom group)are alsothe ones which have the highest selection of skip-connections .
Revised: Figure(second row) also highlights that the worst models (bottom group) also have the highest selection of skip-connections .

Original: Figure 4 further conﬁrms this observations showing how signiﬁcantly worse the models in the search space become as they include  more than 2 skip connections.The trend introduced by linear layers is also visible, but not as much.
Revised: Figure 4 further conﬁrms this observations showing how signiﬁcantly worse the models in the search space become as they include less linear layers, or more than 2 skip connections.

Original: Interestingly, we found that the micro cells in our search space yielding the best validation and test PERs are similar to the structure found in the state-of-the-art convolution models such as time-depthseparable (TDS) models [34].
Revised: Interestingly, we found that the micro cells in our search space yielding the best validation PER is similar to the structure found in the state-of-the-art convolution models such as time-depth-separable (TDS) models [Pratap et al., 2020].

Original: Figure 5 shows the TDS-cell and the micro-cells with best validation and test PERs (from left to right).
Revised: Figure 5 shows the TDS-cell and the micro-cells with best validation and test PERs (from left to right).

Original: This indicates the potential of conducting NAS on smaller datasets to discover new architectures yielding state-of-the-art results on larger datasets.
Revised: This indicates the potential of conducting NAS on smaller datasets to discover new architectures yielding state-of-the-art results on larger datasets.

Original: Latency, Params, FLOPs and PER.
Revised: Latency, Params, FLOPs and PER.

Original: Figure 6 investigates the relationship among test PER, number of parameters and latency of models in the dataset.
Revised: Figure 6 investigates the relationship among test PER, number of parameters and latency of models in the dataset.

Original: FLOPS counts are closely correlated to the number of parameters so are not shown for simplicity.
Revised: FLOPS counts are closely correlated to the number of parameters so are not shown for simplicity.

Original: Although this ﬁgure shows only latency measured on a desktop GPU (NVIDIA GeForce GTX 1080 Ti) with batch size of 32, the dataset includes latency measured on different devices with varying batch sizes.
Revised: Although this ﬁgure shows only latency measured on a desktop GPU (NVIDIA GeForce GTX 1080 Ti) with batch size of 32, the NASBench-ASR includes latency measured on different devices with varying batch sizes.

Original: Also shown in the ﬁgure are 13 Pareto-optimal models of test PER over latency which are indicated by purple (x) markers.
Revised: Also shown in the ﬁgure are 13 Pareto-optimal models, indicated by purple (x) markers.

Original: The best models in terms of test and validation PER are highlighted in red markers.
Revised: Also, the best models on test and validation dataset are highlighted in red markers.

Original: TDS-like models  achieve test PER between 22.7% and 23.45 % and  the best validation PER model has test PER of 22.13%.
Revised: TDS-like models ([Pratap et al., 2020]) achieve test PER between 22 . 7 % and 23 . 45 %, and the best validation PER model has test PER of 22.13%.

Original: In addition, there are Pareto-optimal models that achieve much lower latency without signiﬁcant increase in PER, for instance, the model indicated by violet ( ⬩ ) has one-seventh of the latency of the best test PER model and still has a test PER of 21.92%.
Revised: There are Pareto-optimal models that achieve much lower latency without signiﬁcant increase in PER, for instance, the model indicated by violet ( ⬩ ) has one-ﬁfth of the latency of the best test PER model and still has a test PER of 21.98%.

Original: The distribution of coloured clusters along the y axis suggests that the number of parameters is not a strong factor to determine the accuracy of models.
Revised: The distribution of coloured clusters along the y axis suggests that the number of parameters is not a strong factor to determine the accuracy of models.

Original: Analogically, models with similar number of parameters can lead to very different latency.
Revised: Analogically, models with similar number of parameters can lead to very different latency.

Original:  T RANSFERABILITY
Revised:  T RANSFERABILITY

Original: Training an ASR architecture to achievethe state-of-the-art Word Error Rate (WER) is a slow process, as it often requires iterating over tens of thousand hours of speech data, and potentially over various hyperparameter choices.
Revised: Training an ASR architecture to achieve state-of-the-art WER is a slow process, as it often requires iterating over tens of thousands hours of speech data, and potentially over various hyper-parameter choices.

Original: Even when distributed across multiple GPUs, this process can take days or weeks [1; 17].
Revised: Even when distributed across multiple GPUs, this process can take days or weeks [Amodei et al., 2016; Kahn et al., 2020].

Original: Since the largest strides in lowering WERs have resulted from new architectures being developed manually by speech experts slowly over decades, it would be beneﬁcial to scale such exploration through NAS.
Revised: Since the largest strides in lowering WERs have resulted from new architectures being developed manually by speech experts slowly over decades, it would be beneﬁcial to scale such exploration through NAS.

Original: However, performing NAS directly on large scale ASR datasets needed for peak performance is not practical due to signiﬁcantly more computational costs.
Revised: However, performing NAS directly on large scale ASR datasets needed for peak performance is not practical due to signiﬁcantly more computational costs.

Original: An issue which is further exacerbated when one is also interested in architectures yielding lower latency and computational cost.
Revised: An issue which is further exacerbated when one is also interested in architectures yielding lower latency and computational cost.

Original: For this reason we explore how well new architectures found in NAS search over TIMIT (5 hours) transfer to Librispeech (100 hours).
Revised: For this reason, we explore how well new architectures found by NAS on TIMIT (5 hours) transfer to Librispeech (100 hours).

Original: Speciﬁcally, we investigate whether there is a high correlation between the  best/random/worst performingnew architectures found on TIMIT in terms of PER, to the Label Error Rate (LER) and WER values when training the same architectures on Librispeech.
Revised: Speciﬁcally, we investigate the correlation between the PER of best/random/worst performing architectures found on TIMIT, and the Label Error Rate (LER) (or WER) obtained when training the same architectures on Librispeech.

Original: If a high correlation is present this would mean that one could lower the computational requirements for exploring new architectures in larger datasets byinstead running NAS on that same architecture landscape but with smaller datasets instead.
Revised: If a high correlation is present, this would mean that one could lower the computational requirements of NAS by running it on a smaller datasets.

Original: In our case this would mean a 20x speedup in NAS, and in general this would open a new avenue for research, namely on how much one could save by running NAS on different smaller datasets, or running NAS on subsets of  larger datasets extracted by current or next generation data summarization techniques.
Revised: In our case this would mean a 20x speedup in NAS, and in general this would open a new avenue for research, namely on how much one could save by running NAS on different smaller datasets, or running NAS on a subset of a larger dataset extracted by current or next generation data summarization techniques.

Original: Librispeech Dataset.
Revised: Librispeech Dataset.

Original: Librispeech [30] is a widely used ASR training and benchmarking database of about 1000 hours ofread speech derived from audiobooks.
Revised: Librispeech [Panayotov et al., 2015] is a widely used ASR training and benchmarking database of about 1 , 000 hours of speech derived from audiobooks.

Original: The training dataset is partitioned into clean-100, clean-360 and other-500 sets with the clean partitions exhibiting much higher SNR.
Revised: The training dataset is partitioned into clean-100, clean-360 and other-500 sets with the clean partitions exhibiting much higher SNR.

Original: The test and dev sets are also partitioned similarly.
Revised: The test and dev sets are also partitioned similarly.

Original: In our work, we have used clean-100 partition for training and dev-clean and test-clean for validation and testing.
Revised: In our work, we have used clean100 partition for training and dev-clean and test-clean for validation and testing.

Original: The choice of 100 hours partition is to strike a balance between building a decent large scale model while still keeping the computational cost involved in architecture search experiments plausible.
Revised: The choice ofhours partition is to strike a balance between building a decent large scale model, while still keeping the computational cost involved in architecture search experiments plausible.

Original: Previous literature indicates that the models trained on 100 hours partition are about 1-1.5% absolute [30] behind the models trained on 1000 hours when evaluated on test-clean set.
Revised: Previous literature indicates that the models trained on 100 hours partition are about 1-1.5% absolute [Panayotov et al., 2015] behind the models trained on 1 , 000 hours when evaluated on test-clean set.

Original: Training.
Revised: Training.

Original: Librispeech models were trained with 4 GPUs with Horovod distributed framework.
Revised: Librispeech models were trained with 4 GPUs with Horovod distributed framework.

Original: We use 80 -dimensional log-mel features computed over 25 ms Hamming windows strided by 10ms.
Revised: We use 80 -dimensional log-mel features computed over 25 ms Hamming windows strided by 10ms.

Original: Guided by latency considerations, we used a vocabulary of 780 sub-word tokens as output.
Revised: Guided by latency considerations, we used a vocabulary of 780 sub-word tokens as output.

Original: Model performances are measured by tracking CTC loss, LER, and WER on the validation dataset during training and on test dataset post-training.
Revised: Model performances are measured by tracking CTC loss, LER, and WER on the validation dataset during training and on test dataset post-training.

Original: Pilot Experiments.
Revised: Pilot Experiments.

Original: Similarly to §3.3, we also conducted a pilot experiment to determine appropriate macro structure parameters and hyper-parameters to train the models for Librispeech.
Revised: Similarly to §3.3, we also conducted a pilot experiment to determine appropriate macro structure parameters and hyper-parameters to train the models for Librispeech.

Original: To promote transferability between datasets at a reasonable compute cost, with the micro cell of the best model found on TIMIT dataset (out of 8 , 242 ), we conducted another grid search but instead on
Revised: To promote transferability between datasets at a reasonable compute cost, with the micro cell of the best model found on TIMIT dataset (out of 8 , 242 ), we conducted another grid search but instead on

Original: Librispeech dataset.
Revised: Librispeech dataset.

Original: Accordingly, we chose the ﬁnal number of ﬁlters, kernel size, dilation, learning rate, decay factor and start epoch based on lowest LER/WER on Librispeech.
Revised: Accordingly, we chose the ﬁnal number of ﬁlters, kernel size, dilation, learning rate, decay factor and start epoch based on lowest LER/WER on Librispeech.

Original: Correlation Analysis.
Revised: Correlation Analysis.

Original: For a meaningful analysis we selected 100 architectures from our search space based on their TIMIT PER, namely we took the best 20 , 70 random, and the worst 10 .
Revised: For a meaningful analysis we selected 100 architectures from our search space based on their TIMIT PER, namely we took the best 20 , 70 random, and the worst 10 .

Original: We then plugged the micro cell architecturesin these models into the macro architecturedescribed in the previous subsection for Librispeech.
Revised: We then plugged the micro cell architectures into the macro architecture for Librispeech.

Original: Finally, we trained these models on Librispeech with the hyper-parameters reported earlier and computed their LERs/WERs.
Revised: Finally, we trained these models on Librispeech with the hyper-parameters reported earlier and computed their LERs/WERs.

Original: We observe that models that didn’t converge for TIMIT also didn’t converge for Librispeech.
Revised: We observe that models that didn’t converge for TIMIT also didn’t converge for Librispeech.

Original: For this reason, when looking at the strength of transferability between TIMIT and Librispeech we considered models that had TIMIT PER lower than 50% .
Revised: For this reason, when looking at the strength of transferability between TIMIT and Librispeech we considered models that had TIMIT PER lower than 50% .

Original: As shown in Figure 7, there is a high correlation between TIMIT PER and Librispeech WER.
Revised: As shown in Figure 7, there is a high correlation between TIMIT PER and Librispeech WER.

Original: In particular, we observe 0 .
Revised: In particular, we observe 0 .

Original: 87 correlation between validation metrics, and 0 .
Revised: 85 correlation between validation metrics, and 0 .

Original: 84 for test metrics.
Revised: 86 for test metrics.

Original: This indicates indeed that it is possible to do indirect NAS for cell micro architecture on Librispeech by instead applying NAS directly to TIMIT and then reusing the found cell structures in  macro architectures suitable for larger datasets.
Revised: Thus, it is possible to do indirect NAS for cell micro architecture on Librispeech by ﬁrst applying NAS on TIMIT, and then reusing the identiﬁed cell structures in a macro architecture suitable for Librispeech. [Figure]

Original:  NAS E XPERIMENTS
Revised:  NAS E XPERIMENTS

Original: Followingthe existing  benchmarks, we include results from running different NAS algorithms on our search space, using the proposed dataset to query for accuracy of models if applicable.
Revised: Following existing NAS benchmarks, we include results from running different NAS algorithms on our search space, using the NAS-Bench-ASR to query for accuracy of models.

Original: We consider the following standard NAS algorithms: Regularized Evolution (RE) [36], Hyperband (HYPB) [25], REINFORCE with a LSTM-based controller (REINF) [46], Binary Predictor (BP)[10], and Random Search (RAND) [26].
Revised: We consider the following standard NAS algorithms: Regularized Evolution (RE) [Real et al., 2018], Hyperband (HYPB) [Li et al., 2017], REINFORCE with an LSTM-based controller (REINF) [Zoph & Le, 2017], Binary Predictor (BP)

Revised: [Dudziak et al., 2020], and Random Search (RAND)

Original:  We also include an algorithm based on Deep QLearning (QLRN) which behaves similarly to RE but tries to chain mutations (actions) and learns what mutations can eventually lead to better results.
Revised: [Li & Talwalkar, 2019]. We also include an algorithm based on Deep Q-Learning (QLRN), which behaves similarly to RE but tries to chain mutations (actions) and learns what mutations can eventually lead to better results.

Original: We report results as functions of “trained models” to unify notion of cost.
Revised: We report results as functions of “trained models” to unify notion of cost.

Original: Because in the full training setting we trained models for 40 epochs, if[Figure] an algorithm trains a model for fewer epochs than that (e.g. HYPB)
Revised: Because in the full training setting we trained models for 40 epochs, if an algorithm trains a model for fewer epochs than that (e.g. HYPB)

Original: we consider the cost of such a training to be n / 40 of a “trained model”, where n is the number of epochs used ( 0 < n ≤ 40 ).
Revised: we consider the cost of such a training to be n / 40 of a “trained model”, where n is the number of epochs used ( 0 < n ≤ 40 ).

Original: We run each algorithm 100 times and present results in Figure 8 as average test PER of the best model found (all algorithms were using validation accuracy as their optimization target), and average validation PER of the most recently trained model (after smoothing the curve using exponential moving average with weight 0.9).
Revised: We run each algorithm 100 times and present results in Figure 8 as average test PER of the best model found (all algorithms were using validation accuracy as their optimization target), and average validation PER of the most recently trained model (after smoothing the curve using exponential moving average with weight 0.9).

Original: The ﬁrst observation by looking at the results is that neither of the algorithms was able to consistently ﬁnd the best model – this is somewhat surprising to us considering the moderate size our search space and the fact thatfrom our experience algorithms like RE are able to consistently ﬁnd the best model on  similar benchmarks, e.g., NAS-Bench-201 (given similar budget).
Revised: We found that none of the algorithms were able to consistently ﬁnd the best model – this is somewhat surprising given the moderate size our search space and the fact that algorithms like RE are able to consistently ﬁnd the best model on NAS-Bench-201, given similar budget.

Original: Another interesting observation is related to the weak correlation between validation and test accuracies mentioned in the previous sections.
Revised: Another interesting observation is related to the weak correlation between validation and test accuracies mentioned in the previous sections.

Original: By comparing the ﬁgures on the right and left sides, we can see that algorithms which tend to better optimize their rewards (validation accuracy) as search progresses are not necessarily better in the test PER objective – this is especially visible in the case of REINFORCE, which signiﬁcantly outperforms random search if validation accuracy is considered, but when evaluated using test PER it presents comparable results.
Revised: By comparing the ﬁgures on the right and left sides (Figure 8), we can see that algorithms which tend to better optimize their rewards (validation accuracy) as search progresses are not necessarily better in the test PER objective – this is especially visible in the case of REINFORCE, which signiﬁcantly outperforms random search if validation accuracy is considered, but when evaluated using test PER it presents comparable results.

Original: Another surprise is weak performance of Hyperband – the only algorithm that has the ability to adjust number of epochs to train models.
Revised: Another surprise is weak performance of Hyperband – the only algorithm that has the ability to adjust number of epochs to train models.

Original: From our observations it seems that many of our models converge to relatively good PER values fast (after 5-10 epochs) so we expected Hyperband to be able to leverage this property.
Revised: From our observations it seems that many of our models converge to relatively good PER values fast (after 5-10 epochs) so we expected Hyperband to be able to leverage this property.

Original: However, it seems that in the case of our search space this is not so helpful – potentially because models might still change relative ordering as they are trained for more epochs despite the fact that they have already achieved decent performance.
Revised: However, it seems that in the case of our search space this is not so helpful – potentially because models might still change relative ordering as they are trained for more epochs despite the fact that they have already achieved decent performance.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We introduced NAS-Bench-ASR, a ﬁrst comprehensive dataset for allowing computationally inexpensive training/test time evaluation of model performances, while conducting NAS research in the domain of ASR.
Revised: We introduced NAS-Bench-ASR, a ﬁrst comprehensive dataset for allowing computationally inexpensive training/test time evaluation of model performances, while conducting NAS research in the domain of ASR.

Original: We trained 8 , 242 unique models on the TIMIT dataset across 3 different seeds and in addition to the common training metrics in ASR, e.g., PER, CTC-loss, we also provide information on the number of parameters, FLOPS and latency of running all the models on three different hardware platforms.
Revised: We trained 8 , 242 unique models on the TIMIT dataset for three initializations and three target epochs. In addition to the common training metrics in ASR, e.g., PER, CTC-loss, we also provide information on the number of parameters, FLOPS and latency of running all the models on two hardware platforms for varying batch sizes.

Original: Model run-times are especially beneﬁcial for NAS researchers interested in optimizing architectures for efﬁcient on-device deployment.
Revised: Model run-times are especially beneﬁcial for NAS researchers interested in optimizing architectures for efﬁcient on-device deployment.

Original: We presented a comprehensive analysis of our dataset and evaluated the performances of a number of NAS algorithms on it.
Revised: We presented a comprehensive analysis of our dataset and evaluated the performances of a number of NAS algorithms on it.

Original: Lastly, we show that good cell structures as identiﬁed on the TIMIT dataset transfer well to Librispeech, which paves the way for affordable NAS on ASR domain. 
Revised: Lastly, we show that good cell structures as identiﬁed on the TIMIT dataset transfer well to Librispeech, which paves the way for affordable NAS on ASR domain. A A PPENDIX

Revised: Our dataset comprises of 8,242 unique models that were trained for three target epochs and each starting with three different initializations.

Revised: In this appendix we present the results for the models with different initializations and target epochs.

Revised: A. C ORRELATION BETWEEN V ALIDATION AND T EST PER S FOR THE RUNS WITH DIFFERENT TARGET EPOCHS

Revised: In this section we present the correlation of validation and test PERs between different target epochs.

Revised: Speciﬁcally, we correlate each run of target epoch 40 (i.e., runs starting with unique initialization and trained for 40 epochs) with the runs for target epoch 5 and 10 .

Revised: This analysis informs us if reduced training (i.e., target epoch of 5 or 10 ) can be used as a good proxy for NAS.

Revised: As shown in

Revised: Table 3, there is weak correlation of target epoch 40 and 5 .

Revised: However, there is a moderate correlation between target epoch 40 and 10 , suggesting that the reduced training might be useful for performing

Revised: NAS.

Revised: [Table]

Revised: A. C ORRELATION BETWEEN V ALIDATION AND T EST PER S

Revised: In this section we present the correlation between the ﬁnal test PER and the best validation PER for the three runs for target epoch 40 starting with different initializations, which is extension of results presented in Figure 2.

Revised: As shown in Figure 9, the results for all seeds are similar.

Revised: Speciﬁcally, there is moderate correlation for top models, weak for middle ones, and strong for worst ones.

Revised: A. C ELL ARCHITECTURES IN OUR SEARCH SPACE

Revised: In this section we present the micro-cell architectures in our search space that achieved best validation and test PERs for the three runs starting with different initializations.

Revised: Figure 10 shows the micro-cell architectures that achieved best validation PERs (in top row) and best test PERs (in bottom row).

Revised: We observe that the micro-cell architectures of top models often start with two convolution layers followed by a linear layer (which is consistent across all architectures).

Revised: A. I MPACT OF DIFFERENT OPERATIONS - COUNT ON MODEL PERFORMANCE

Revised: In this section we present the impact of the number of selected operation on the model performance.

Revised: As shown in Figure 11, models with more convolution layers, or with no linear layer are ranked as worst performing models.

Revised: At the same time, models with a zero ops in the main edge have poor performance.

Revised: This indicates that the top performing models are designed such that all nodes comprise of linear and convolution layers.

Revised: We also observe that an increase in the number of skip connections deteriorates performance signiﬁcantly, with surprisingly two skip connections yielding similar results to the traditional one skip connection, which aims to learn a perturbation of the identity rather than a generic transformation between input and output nodes.

Revised: [Figure]