Original: R ES P ERF N ET
Revised: R ES P ERF N ET

Original: : D EEP R ESIDUAL L EARNING FOR R EGRESSIONAL
Revised: : D EEP R ESIDUAL L EARNING FOR R EGRESSIONAL

Original: P
Revised: P

Original: ERFORMANCE
Revised: ERFORMANCE

Original: M ODELING
Revised: M ODELING

Original: OF D EEP N EURAL N ETWORKS
Revised: OF D EEP N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The rapid advancements of computing technology facilitate the development of diverse deep learning applications.
Revised: The rapid advancements of computing technology facilitate the development of diverse deep learning applications.

Original: Unfortunately, the efﬁciency of parallel computing infrastructures varies widely with neural network models, which hinders the exploration of the design space to ﬁnd high-performance neural network architectures on speciﬁc computing platforms for a given application.
Revised: Unfortunately, the efﬁciency of parallel computing infrastructures varies widely with neural network models, which hinders the exploration of the design space to ﬁnd high-performance neural network architectures on speciﬁc computing platforms for a given application.

Original: To address such a challenge, we propose a deep learning-based method, ResPerfNet , which trains a residual neural network with representative datasets obtained on the target platform to predict the performance for a deep neural network.
Revised: To address such a challenge, we propose a deep learning-based method, ResPerfNet , which trains a residual neural network with representative datasets obtained on the target platform to predict the performance for a deep neural network.

Original: Our experimental results show that ResPerfNet can accurately predict the execution time of individual neural network layers and full network models on a variety of platforms.
Revised: Our experimental results show that ResPerfNet can accurately predict the execution time of individual neural network layers and full network models on a variety of platforms.

Original: In particular, ResPerfNet achieves 8.4% of mean absolute percentage error for
Revised: In particular, ResPerfNet achieves 8.4% of mean absolute percentage error for

Original: LeNet, AlexNet and VGG16 on the NVIDIA GTX 1080Ti, which is substantially lower than the previously published works.
Revised: LeNet, AlexNet and VGG16 on the NVIDIA GTX 1080Ti, which is substantially lower than the previously published works.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning (DL) has exploded successfully and is applied to many application domains, such as image recognition and object detection
Revised: Deep learning (DL) has exploded successfully and is applied to many application domains, such as image recognition and object detection

Original: Thus, a lot of human experts design high-accuracy neural network architectures for different applications.
Revised: Thus, a lot of human experts design high-accuracy neural network architectures for different applications.

Original: However, for Internet of Things (IoT) applications, large neural network models cannot ﬁt into resource-constrained devices.
Revised: However, for Internet of Things (IoT) applications, large neural network models cannot ﬁt into resource-constrained devices.

Original: On the other hand, a system designer often tries to ﬁnd a proper computing platform or a deep learning accelerator (DLA) to execute a DL application with acceptable responsiveness.
Revised: On the other hand, a system designer often tries to ﬁnd a proper computing platform or a deep learning accelerator (DLA) to execute a DL application with acceptable responsiveness.

Original: An exhaustive way to optimize the system design is to evaluate the cost and performance of desired DL models on all the available hardware/software options, but it is not only tedious but costly and lengthy in practice.
Revised: An exhaustive way to optimize the system design is to evaluate the cost and performance of desired DL models on all the available hardware/software options, but it is not only tedious but costly and lengthy in practice.

Original: Since DL frameworks and accelerators are evolving rapidly, and even some slight changes could signiﬁcantly impact the performance of DL applications, it may be necessary to update the performance models frequently.
Revised: Since DL frameworks and accelerators are evolving rapidly, and even some slight changes could signiﬁcantly impact the performance of DL applications, it may be necessary to update the performance models frequently.

Original: Therefore, we need a systematic and efﬁcient approach to produce accurate performance models when changes occur.
Revised: Therefore, we need a systematic and efﬁcient approach to produce accurate performance models when changes occur.

Original: While several works (Qi et al.
Revised: While several works (Qi et al.

Original: ; Justus et al.
Revised: ; Justus et al.

Original: Wang et al.) have been proposed to estimate the delivered performance of a given DL model on a speciﬁc computing platform, so as to rapidly evaluate design alternatives, the estimates from these efforts are not very accurate.
Revised: Wang et al.) have been proposed to estimate the delivered performance of a given DL model on a speciﬁc computing platform, so as to rapidly evaluate design alternatives, the estimates from these efforts are not very accurate.

Original: For example, the mean absolute percentage error (MAPE) for estimating full neural network models such as LeNet (LeCun et al.
Revised: For example, the mean absolute percentage error (MAPE) for estimating full neural network models such as LeNet (LeCun et al.

Original: (1998)), AlexNet (Krizhevsky et al.
Revised: (1998)), AlexNet (Krizhevsky et al.

Original: and
Revised: and

Original: VGG16 (Simonyan & Zisserman) on the NVIDIA GTX 1080Ti is as high as 24% in Wang et al., whose accuracy is the best among the previous works, but still has room for improvement.
Revised: VGG16 (Simonyan & Zisserman) on the NVIDIA GTX 1080Ti is as high as 24% in Wang et al., whose accuracy is the best among the previous works, but still has room for improvement.

Original: In this paper, we propose a deep residual network architecture, called ResPerfNet, to efﬁciently and accurately model the performance of DL models running on a wide range of DL frameworks and
Revised: In this paper, we propose a deep residual network architecture, called ResPerfNet, to efﬁciently and accurately model the performance of DL models running on a wide range of DL frameworks and

Original: DLAs.
Revised: DLAs.

Original: It is based on the residual function approach proposed by (He et al.
Revised: It is based on the residual function approach proposed by (He et al.

Original: and inspired by the prior works Liu & Yang (2018); Jha et al.
Revised: and inspired by the prior works Liu & Yang (2018); Jha et al.

Original: ; Wan et al.
Revised: ; Wan et al.

Original: (2019)), which use residual neural networks to solve regression problems.
Revised: (2019)), which use residual neural networks to solve regression problems.

Original: The proposed model can be trained with performance data collected from many system conﬁgurations to establish a uniﬁed performance predictor which assists the users in selecting the DL model, the DL framework, and the DLA for their applications.
Revised: The proposed model can be trained with performance data collected from many system conﬁgurations to establish a uniﬁed performance predictor which assists the users in selecting the DL model, the DL framework, and the DLA for their applications.

Original: Extensive experiments have been done to show that our uniﬁed approach not only provides more accurate performance estimates than the previous works, but also enables the users to quickly pre- dict the performance of their DL applications executed with various models-framework-accelerator conﬁgurations.
Revised: Extensive experiments have been done to show that our uniﬁed approach not only provides more accurate performance estimates than the previous works, but also enables the users to quickly pre- dict the performance of their DL applications executed with various models-framework-accelerator conﬁgurations.

Original: The contributions of this paper are summarized as follows.
Revised: The contributions of this paper are summarized as follows.

Original: • An uniﬁed DL-based approach for estimating the computing performance of DL applications on a variety of models-framework-accelerator conﬁgurations, which enables the users to explore the hardware/software design space quickly.
Revised: • An uniﬁed DL-based approach for estimating the computing performance of DL applications on a variety of models-framework-accelerator conﬁgurations, which enables the users to explore the hardware/software design space quickly.

Original: • A novel deep residual neural architecture is proposed to deliver the most accurate performance predictions that we are aware of.
Revised: • A novel deep residual neural architecture is proposed to deliver the most accurate performance predictions that we are aware of.

Original: Experimental results conﬁrm that our approach yields lower prediction errors on across various platforms.
Revised: Experimental results conﬁrm that our approach yields lower prediction errors on across various platforms.

Original: The remaining of this paper is organized as follows.
Revised: The remaining of this paper is organized as follows.

Original: Section 2 presents the related work.
Revised: Section 2 presents the related work.

Original: Section 3 describes the architecture of ResPerfNet.
Revised: Section 3 describes the architecture of ResPerfNet.

Original: Section 4 shows the proposed systematic modeling method.
Revised: Section 4 shows the proposed systematic modeling method.

Original: Section 5 elaborates the dataset and training mechanism to train the ResPerfNet models within a reasonable time span.
Revised: Section 5 elaborates the dataset and training mechanism to train the ResPerfNet models within a reasonable time span.

Original: Section 6 evaluates the efﬁciency of our approach.
Revised: Section 6 evaluates the efﬁciency of our approach.

Original: Section 7 concludes the paper.
Revised: Section 7 concludes the paper.

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original: With the rapid evolving of both hardware accelerators and DL models, the performance measure/estimation of the DL models on the DLA platforms is an important task to evaluate the effectiveness of the software/hardware solutions to the given problems.
Revised: With the rapid evolving of both hardware accelerators and DL models, the performance measure/estimation of the DL models on the DLA platforms is an important task to evaluate the effectiveness of the software/hardware solutions to the given problems.

Original: Different approaches have been proposed to serve the purposes.
Revised: Different approaches have been proposed to serve the purposes.

Original: Benchmarking approaches , such as DAWNbench (Coleman et al.
Revised: Benchmarking approaches , such as DAWNbench (Coleman et al.

Original: and MLPerf (Reddi et al.
Revised: and MLPerf (Reddi et al.

Original: (2020)), aim at the measurements of the training and inference performance of the machine-learning (ML) models on certain software/hardware combinations.
Revised: (2020)), aim at the measurements of the training and inference performance of the machine-learning (ML) models on certain software/hardware combinations.

Original: By offering a set of standardized machine learning workloads and the instructions for performance benchmarking, these benchmarks are able to measure how fast a system can perform the training and inference for ML models.
Revised: By offering a set of standardized machine learning workloads and the instructions for performance benchmarking, these benchmarks are able to measure how fast a system can perform the training and inference for ML models.

Original: Analytical approach , as reported in PALEO (Qi et al.), constructs the analytical performance model for DL systems.
Revised: Analytical approach , as reported in PALEO (Qi et al.), constructs the analytical performance model for DL systems.

Original: The execution time is decomposed into the total time for the computation and communication parts, which are derived from the utilization of the computing and communication resources on the target hardware, respectively.
Revised: The execution time is decomposed into the total time for the computation and communication parts, which are derived from the utilization of the computing and communication resources on the target hardware, respectively.

Original: For instance, the computation time is estimated by dividing the total ﬂoating-point operations required by the DL model to the actual processing speed (i.e., the processed ﬂoating-point operations per second for the DL model) delivered by the computing hardware.
Revised: For instance, the computation time is estimated by dividing the total ﬂoating-point operations required by the DL model to the actual processing speed (i.e., the processed ﬂoating-point operations per second for the DL model) delivered by the computing hardware.

Original: The communication time is calculated by the similar approach.
Revised: The communication time is calculated by the similar approach.

Original: This approach highly relies on the accuracy of the benchmarking results (i.e., to provide the actual processing speed of the target model on the hardware), which requires its users to choose the benchmarks wisely to perfectly match the program characteristics of their target deep learning models, so as to give a proper estimate of the actual processing speed.
Revised: This approach highly relies on the accuracy of the benchmarking results (i.e., to provide the actual processing speed of the target model on the hardware), which requires its users to choose the benchmarks wisely to perfectly match the program characteristics of their target deep learning models, so as to give a proper estimate of the actual processing speed.

Original: However, the manual process (of the benchmarks selection) limit its widespread adoption.
Revised: However, the manual process (of the benchmarks selection) limit its widespread adoption.

Original: DL-based approaches build the DNNs for estimating the DL models’ performance by learning the relationships between the characteristics of the DL models and the speciﬁcations of the accelerating hardware.
Revised: DL-based approaches build the DNNs for estimating the DL models’ performance by learning the relationships between the characteristics of the DL models and the speciﬁcations of the accelerating hardware.

Original: The following works focus on TensorFlow-based DL models.
Revised: The following works focus on TensorFlow-based DL models.

Original: Justus et al.
Revised: Justus et al.

Original: (2018) use a fully-connected multiple-layer perceptron (MLP) network for performance prediction, using the conﬁgurations of the DL model and the speciﬁcation of the hardware accelerator, and the training data of the DL model as the input features to the MLP network.
Revised: (2018) use a fully-connected multiple-layer perceptron (MLP) network for performance prediction, using the conﬁgurations of the DL model and the speciﬁcation of the hardware accelerator, and the training data of the DL model as the input features to the MLP network.

Original: However, due to the simpliﬁed communication time estimation model, where the communications from GPU to CPU for each of the DL layers are counted repeatedly for estimating the communication time, their model tends to provide over-estimated results.
Revised: However, due to the simpliﬁed communication time estimation model, where the communications from GPU to CPU for each of the DL layers are counted repeatedly for estimating the communication time, their model tends to provide over-estimated results.

Original: Wang et al.
Revised: Wang et al.

Original: use PerfNet (an MLP network) to learn the relationships between the conﬁgurations and the execution time of the target DL model.
Revised: use PerfNet (an MLP network) to learn the relationships between the conﬁgurations and the execution time of the target DL model.

Original: They further decompose the execution of a DL model into three phases, preprocessing, execution, and postprocessing, and train multiple PerfNet network instances, each of which learns the relationships between the model conﬁgurations and the model execution time for a speciﬁc phase.
Revised: They further decompose the execution of a DL model into three phases, preprocessing, execution, and postprocessing, and train multiple PerfNet network instances, each of which learns the relationships between the model conﬁgurations and the model execution time for a speciﬁc phase.

Original: By aggregating the prediction results for the three phases, their proposed work is able to predict the total execution time of a given
Revised: By aggregating the prediction results for the three phases, their proposed work is able to predict the total execution time of a given

Original: DL model.
Revised: DL model.

Original: Nevertheless, the MLP network has its own limitation, i.e., it is hard to further enhance its performance since a deeper MLP network will lead to lower prediction accuracy.
Revised: Nevertheless, the MLP network has its own limitation, i.e., it is hard to further enhance its performance since a deeper MLP network will lead to lower prediction accuracy.

Original: In consideration of the limitations of the prior works listed above and the need of modeling the optimizing DL frameworks, our work uses the systematical approach to characterize the DL models built with various DL framework, and adopts the residual neural network to model their delivered performance on the DLAs.
Revised: In consideration of the limitations of the prior works listed above and the need of modeling the optimizing DL frameworks, our work uses the systematical approach to characterize the DL models built with various DL framework, and adopts the residual neural network to model their delivered performance on the DLAs.

Original:  R ES P ERF N ET A RCHITECTURE
Revised:  R ES P ERF N ET A RCHITECTURE

Original: ResPerfNet adopts a ML-based approach for the performance estimation of different types of neural network layers.
Revised: ResPerfNet adopts a ML-based approach for the performance estimation of different types of neural network layers.

Original: Furthermore, ResPerfNet is specially designed to prevent the degradation problem, which refers to the phenomenon that increasing the depth and/or the width of each layer for the DNN may not only necessarily improve the accuracy, but get saturated rapidly and then degrades sharply as reported in (He & Sun (2015); Srivastava et al.
Revised: Furthermore, ResPerfNet is specially designed to prevent the degradation problem, which refers to the phenomenon that increasing the depth and/or the width of each layer for the DNN may not only necessarily improve the accuracy, but get saturated rapidly and then degrades sharply as reported in (He & Sun (2015); Srivastava et al.

Original: In other words, it is more likely to lead to a higher training error on the neural network with a wider or deeper architecture.
Revised: In other words, it is more likely to lead to a higher training error on the neural network with a wider or deeper architecture.

Original: To solve the problem, the deep residual learning is proposed and applied to each group of the stacked NN layers (He et al.
Revised: To solve the problem, the deep residual learning is proposed and applied to each group of the stacked NN layers (He et al.

Original: (2016)), where a certain number of stacked layers are logically grouped together to form a residual block .
Revised: (2016)), where a certain number of stacked layers are logically grouped together to form a residual block .

Original: Hence, in this work, to address the degradation problem, we adopt the deep residual learning to every few stacked layers (He et al.
Revised: Hence, in this work, to address the degradation problem, we adopt the deep residual learning to every few stacked layers (He et al.

Original: The residual block is deﬁned as Equation 1, where x and y represent the input feature maps and the output vectors of the residual layer, respectively.
Revised: The residual block is deﬁned as Equation 1, where x and y represent the input feature maps and the output vectors of the residual layer, respectively.

Original: The function F ( x , { W i } ) performs the residual operations to be learned.
Revised: The function F ( x , { W i } ) performs the residual operations to be learned.

Original: The operation F ( x , { W i } ) + x is performed by a shortcut connection and element-wise addition.
Revised: The operation F ( x , { W i } ) + x is performed by a shortcut connection and element-wise addition.

Original: Figure 1 illustrates the network architecture of ResPerfNet.
Revised: Figure 1 illustrates the network architecture of ResPerfNet.

Original: The second, third and fourth layers (i.e., two convolutional and one add layers) together form a residual block, and there are a total of six residual blocks in ResPerfNet.
Revised: The second, third and fourth layers (i.e., two convolutional and one add layers) together form a residual block, and there are a total of six residual blocks in ResPerfNet.

Original: [Equation]
Revised: [Equation]

Original: As shown in Figure 1, the ResPerfNet consists of 26 layers, including 15 convolutional layers, 6 add layers, 4 fully-connected (FC) layers and 1 dropout layer.
Revised: As shown in Figure 1, the ResPerfNet consists of 26 layers, including 15 convolutional layers, 6 add layers, 4 fully-connected (FC) layers and 1 dropout layer.

Original: Before FC layers, every 7 layers contain one head convolutional layer (e.g., Conv1D 3 representing the head convolutional layer for the ﬁrst residual block) and two residual blocks, each of which consists of two convolutional layers with the same ﬁlters and an element-wise add residual function layer.
Revised: Before FC layers, every 7 layers contain one head convolutional layer (e.g., Conv1D 3 representing the head convolutional layer for the ﬁrst residual block) and two residual blocks, each of which consists of two convolutional layers with the same ﬁlters and an element-wise add residual function layer.

Original: The ﬁrst head convolutional layer has 128 ﬁlters of kernel size 3 with a stride length of 1.
Revised: The ﬁrst head convolutional layer has 128 ﬁlters of kernel size 3 with a stride length of 1.

Original: In order to reduce the complexity of
Revised: In order to reduce the complexity of

Original: ResPerfNet, the second head convolutional layer uses 64 ﬁlters of kernel size 3 with a stride length of 1.
Revised: ResPerfNet, the second head convolutional layer uses 64 ﬁlters of kernel size 3 with a stride length of 1.

Original: Moreover, the number of kernels for the six residual blocks is decreasing from 128 kernels in the ﬁrst two blocks to 32 kernels for the last two blocks.
Revised: Moreover, the number of ﬁlters for the six residual blocks is decreasing from 128 ﬁlters in the ﬁrst two blocks to 32 ﬁlters for the last two blocks.

Original: Three FC layers are attached to the last residual block, where each of the FC layers has 128 neurons.
Revised: Three FC layers are attached to the last residual block, where each of the FC layers has 128 neurons.

Original: The dropout layer with the ratio of 0.2 is connected to the last FC layer, which uses a single neuron to perform the one-dimensional regression for predicting the elapsed time of the designated type of the layers.
Revised: The dropout layer with the ratio of 0.2 is connected to the last FC layer, which uses a single neuron to perform the one-dimensional regression for predicting the elapsed time of the designated type of the layers.

Original: Our proposed residual neural architecture, ResPerfNet, gets signiﬁcant improvements in accuracy compared with traditional machine learning algorithms, such as support vector regression, polynomial regression and XGBoost, and is even better than the MLP network.
Revised: Our proposed residual neural architecture, ResPerfNet, gets signiﬁcant improvements in accuracy compared with traditional machine learning algorithms, such as support vector regression, polynomial regression and XGBoost, and is even better than the MLP network.

Original: A series of experiments has been done to show ResPerfNet is superior to the previous works in Section 6.1.
Revised: A series of experiments has been done to show ResPerfNet is superior to the previous works in Section 6.1.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: This section presents the methodology of using ResPerfNet to relate the performance characteristics of a CNN layer to the delivered performance of the given layer.
Revised: This section presents the methodology of using ResPerfNet to relate the performance characteristics of a CNN layer to the delivered performance of the given layer.

Original: We ﬁrst deﬁne the target neural networks for the performance modeling in Section 4.1.
Revised: We ﬁrst deﬁne the target neural networks for the performance modeling in Section 4.1.

Original: The three-phase based modeling of a given
Revised: The three-phase based modeling of a given

Original: CNN based is presented in Section 4.2.
Revised: CNN based is presented in Section 4.2.

Original: Lastly, the same modeling for a given NN layer is further described in Section 4.3.
Revised: Lastly, the same modeling for a given NN layer is further described in Section 4.3.

Original: 4. F ORMALIZING THE N EURAL N ETWORKS
Revised: 4. F ORMALIZING THE N EURAL N ETWORKS

Original: A neural network can be represented by a directed acyclic graph, denoted as N ( { u ( i ) } ki =1 ) , consisting of an ordered sequence of k nodes, where each graph node u ( i ) represents a layer of the neural network N , such as convolutional, pooling, and fully-connected layers.
Revised: A neural network can be represented by a directed acyclic graph, denoted as N ( { u ( i ) } ki =1 ) , consisting of an ordered sequence of k nodes, where each graph node u ( i ) represents a layer of the neural network N , such as convolutional, pooling, and fully-connected layers.

Original: The input and output feature maps of a graph node u ( i ) performing the operation f
Revised: The input and output feature maps of a graph node u ( i ) performing the operation f

Original: ( i ) are denoted as input ( f ( i ) ) and output ( f ( i ) ) , respectively.
Revised: ( i ) are denoted as input ( f ( i ) ) and output ( f ( i ) ) , respectively.

Original: In this work, we assume that a given neural network will be run on the host system h with a single hardware accelerating device d .
Revised: In this work, we assume that a given neural network will be run on the host system h with a single hardware accelerating device d .

Original: 4. T HE T HREE -P HASE P ERFORMANCE M ODELING
Revised: 4. T HE T HREE -P HASE P ERFORMANCE M ODELING

Original: The execution time of a given neural network model includes the computation time spent on the acceleration device d and the data communication time between the host system h and the device d .
Revised: The execution time of a given neural network model includes the computation time spent on the acceleration device d and the data communication time between the host system h and the device d .

Original: As most of the computations are performed by the accelerating device and the communications occur merely at the ﬁrst and the last layers of the given model, the estimated execution time of a given neural network model with k layers is formulated as follows, where the formulation assumes that all k layers within the given model are accelerated by the single device d .
Revised: As most of the computations are performed by the accelerating device and the communications occur merely at the ﬁrst and the last layers of the given model, the estimated execution time of a given neural network model with k layers is formulated as follows, where the formulation assumes that all k layers within the given model are accelerated by the single device d .

Original: [Equation]
Revised: [Equation]

Original: The above equation shows the three-phase performance modeling approach, where T pre , T exe , and
Revised: The above equation shows the three-phase performance modeling approach, where T pre , T exe , and

Original: T post represent the execution time for the preprocess , execution , and postprocess phases, respectively.
Revised: T post represent the execution time for the preprocess , execution , and postprocess phases, respectively.

Original: Speciﬁcally, the communication time of bringing the input data from the host system to the accelerating device at the ﬁrst layer is denoted as T pre ( u (1) ) , where the i -th NN layer is represented as u ( i ) .
Revised: Speciﬁcally, the communication time of bringing the input data from the host system to the accelerating device at the ﬁrst layer is denoted as T pre ( u (1) ) , where the i -th NN layer is represented as u ( i ) .

Original: The summation of the execution time for all the NN layers is represented as (cid:80) ki
Revised: The summation of the execution time for all the NN layers is represented as (cid:80) ki

Original: =1 T exe ( u ( i ) ) .
Revised: =1 T exe ( u ( i ) ) .

Original: The communication time of transferring the inference results from the accelerating device to the host system is deﬁned as T post ( u ( k ) ) .
Revised: The communication time of transferring the inference results from the accelerating device to the host system is deﬁned as T post ( u ( k ) ) .

Original: Our prediction model delivers more accurate performance estimates than previously proposed methods by modeling these three phases deﬁned in the following subsection for a DLA separately and adding the predicted results together as Equation 2.
Revised: Our prediction model delivers more accurate performance estimates than previously proposed methods by modeling these three phases deﬁned in the following subsection for a DLA separately and adding the predicted results together as Equation 2.

Original: 4. M ODELING I NDIVIDUAL NN L AYERS
Revised: 4. M ODELING I NDIVIDUAL NN L AYERS

Original: The similar approach is used to model the performance of the i -th NN layer u ( i ) .
Revised: The similar approach is used to model the performance of the i -th NN layer u ( i ) .

Original: In particular, for each layer u
Revised: In particular, for each layer u

Original: ( i ) , the execution times for the preprocess, execution, and postprocess phases are
Revised: ( i ) , the execution times for the preprocess, execution, and postprocess phases are

Original: T pre ( u ( i ) ) , T exe ( u ( i ) ) , and T post ( u ( i ) ) , respectively.
Revised: T pre ( u ( i ) ) , T exe ( u ( i ) ) , and T post ( u ( i ) ) , respectively.

Original: The above time components constitute the estimated execution time of the layer u
Revised: The above time components constitute the estimated execution time of the layer u

Original: ( i ) , as deﬁned in the equation below.
Revised: ( i ) , as deﬁned in the equation below.

Original: The superscript index i is omitted to simplify the looks of the equations by using the simpler form u .
Revised: The superscript index i is omitted to simplify the looks of the equations by using the simpler form u .

Original: [Equation]
Revised: [Equation]

Original: The preprocess phase is for preparing the input data for the acceleration in d and involves with the four operations: 1) issuing the commands for copying input feature maps on h and d asynchronously, 2) performing the memory copy of the input feature maps in 1, 3) issuing the commands for the operation f on d , and 4) performing the data reshaping operations for input feature maps.
Revised: The preprocess phase is for preparing the input data for the acceleration in d and involves with the four operations: 1) issuing the commands for copying input feature maps on h and d asynchronously, 2) performing the memory copy of the input feature maps in 1, 3) issuing the commands for the operation f on d , and 4) performing the data reshaping operations for input feature maps.

Original: The data reshaping operations, which transform the input/output data to the more efﬁcient format for the next operation on d , usually occur in data transmissions between h and d .
Revised: The data reshaping operations, which transform the input/output data to the more efﬁcient format for the next operation on d , usually occur in data transmissions between h and d .

Original: The lengths of time for the four operations are R ( input ( f ) , h, d ) , M ( input ( f ) , R ( f, d ) , and T ( input ( f ) , d ) , respectively.
Revised: The lengths of time for the four operations are R ( input ( f ) , h, d ) , M ( input ( f ) , R ( f, d ) , and T ( input ( f ) , d ) , respectively.

Original: As shown in Equation 4, the time consumed in the preprocess phase is deﬁned as the summation of the time required by the above four operations.
Revised: As shown in Equation 4, the time consumed in the preprocess phase is deﬁned as the summation of the time required by the above four operations.

Original: [Equation]
Revised: [Equation]

Original: Intuitively, the time consumed  in the execution phase would be identical to the computation time of f  .
Revised: Intuitively, the time consumed for computation, which is C ( f, d ) , in the execution phase would be identical to the computation time of f on d .

Original: Unfortunately, the measured execution time of a layer from the micro-benchmarks includes the time consumed by the data reshaping operations in both directions, from h to d and from d to h , which are T ( input ( f ) , d ) and T ( output ( f ) , d ) , respectively.
Revised: Unfortunately, the measured execution time of a layer from the micro-benchmarks includes the time consumed by the data reshaping operations in both directions, from h to d and from d to h , which are T ( input ( f ) , d ) and T ( output ( f ) , d ) , respectively.

Original: As the deployed NN layers collectively run on the acceleration device d , isolating the data reshaping time from the measured execution time for the NN layer of each micro-benchmark facilitates the execution time estimation of the deployed NN layers with the formula, (cid:80) ki =1 T exe ( u ( i ) ) .
Revised: As the deployed NN layers collectively run on the acceleration device d , isolating the data reshaping time from the measured execution time for the NN layer of each micro-benchmark facilitates the execution time estimation of the deployed NN layers with the formula, (cid:80) ki =1 T exe ( u ( i ) ) .

Original: Regarding this situation, the time for the execution phase is deﬁned in Equation 5.
Revised: Regarding this situation, the time for the execution phase is deﬁned in Equation 5.

Original: [Equation]
Revised: [Equation]

Original: The postprocess phase is deﬁned for dealing with the procedure of returning the inference computation results back to the invoking application on the host system.
Revised: The postprocess phase is deﬁned for dealing with the procedure of returning the inference computation results back to the invoking application on the host system.

Original: That is, it is about reshaping the output vector into the format accepted by h , copying the output vector back to h from d , and moving the prediction result to the application level (i.e., the call site of the model inference) on the host system.
Revised: That is, it is about reshaping the output vector into the format accepted by h , copying the output vector back to h from d , and moving the prediction result to the application level (i.e., the call site of the model inference) on the host system.

Original: The corresponding execution time for the above three operations are denoted as
Revised: The corresponding execution time for the above three operations are denoted as

Original: T ( output ( f ) , d ) , M ( output ( f, d, h ) , and V ( output ( f ) , h ) , respectively.
Revised: T ( output ( f ) , d ) , M ( output ( f, d, h ) , and V ( output ( f ) , h ) , respectively.

Original: [Equation]
Revised: [Equation]

Original:  T RAINING D ATA AND L OSS F UNCTION
Revised:  T RAINING D ATA AND L OSS F UNCTION

Original: In this section, we present the details of the dataset used to build the proposed performance prediction models.
Revised: In this section, we present the details of the dataset used to build the proposed performance prediction models.

Original: In particular, the conﬁgurations of our developed benchmark tools for the training dataset is discussed in Section 5.1.
Revised: In particular, the conﬁgurations of our developed benchmark tools for the training dataset is discussed in Section 5.1.

Original: The tool collecting and extracting the data is described in Section 5.2, and the data transformation techniques to facilitate the training convergence is introduced in Section 5.3.
Revised: The tool collecting and extracting the data is described in Section 5.2, and the data transformation techniques to facilitate the training convergence is introduced in Section 5.3.

Original: The specially designed loss function to better deal with the unbalanced training data is introduced in Section 5.4.
Revised: The specially designed loss function to better deal with the unbalanced training data is introduced in Section 5.4.

Original: 5. D ATA P REPARATION
Revised: 5. D ATA P REPARATION

Original: The training data is the characteristics of the TensorFlow and TensorRT programs and the performance information of the programs running on the target computing hardware, where the proposed model helps correlate the characteristics and their runtimes during the training process.
Revised: The training data is the characteristics of the TensorFlow and TensorRT programs and the performance information of the programs running on the target computing hardware, where the proposed model helps correlate the characteristics and their runtimes during the training process.

Original: In order to better catch the characteristics of different TensorFlow and TensorRT conﬁgurations (i.e., the code patterns, which are considered as the features during the model training process), we have developed a benchmark tool to generate a set of micro-benchmarks, which are actually TensorFlow and
Revised: In order to better catch the characteristics of different TensorFlow and TensorRT conﬁgurations (i.e., the code patterns, which are considered as the features during the model training process), we have developed a benchmark tool to generate a set of micro-benchmarks, which are actually TensorFlow and

Original: TensorRT programs with different conﬁgurations for the three types of layers, including convolution, pooling and dense layers.
Revised: TensorRT programs with different conﬁgurations for the three types of layers, including convolution, pooling and dense layers.

Original: The generation of the micro-benchmarks are done by randomly selecting the conﬁgurations for each type of the layer, so as to collect the performance for different conﬁgurations.
Revised: The generation of the micro-benchmarks are done by randomly selecting the conﬁgurations for each type of the layer, so as to collect the performance for different conﬁgurations.

Original: The possible conﬁgurations (or features) for all three layer types and their ranges are listed in Table 3.
Revised: The possible conﬁgurations (or features) for all three layer types and their ranges are listed in Table 3.

Original: These conﬁgurations are actually the function parameters for the three types of layers, which are extracted from TensorFlow 1.13 APIs, including tensorflow.layers.conv2d , tensorflow.layers.maxpooling2d , and tensorflow.layers.dense , and their possible combinations are 7 .
Revised: These conﬁgurations are actually the function parameters for the three types of layers, which are extracted from TensorFlow 1.13 APIs, including tensorflow.layers.conv2d , tensorflow.layers.maxpooling2d , and tensorflow.layers.dense , and their possible combinations are 7 .

Original: 33 × 10 14 , 7 .
Revised: 33 × 10 14 , 7 .

Original: 33 × 10 10 , and 2 .
Revised: 33 × 10 10 , and 2 .

Original: 14 × 10 9 , respectively.
Revised: 14 × 10 9 , respectively.

Original: While each microbenchmark takes at least seconds for the stable and accurate measurements, it is impossible to cover the entire design space with brute force, which requires over 10 14 micro-benchmark runs.
Revised: While each microbenchmark takes at least seconds for the stable and accurate measurements, it is impossible to cover the entire design space with brute force, which requires over 10 14 micro-benchmark runs.

Original: 5. D ATA C OLLECTION AND D ATA E XTRACTION
Revised: 5. D ATA C OLLECTION AND D ATA E XTRACTION

Original: The data preparation is used to generate the TensorFlow- and TensorRT-based micro-benchmarks.
Revised: The data preparation is used to generate the TensorFlow- and TensorRT-based micro-benchmarks.

Original: The data collection takes about two weeks for running 100,000 different samples of the TensorFlow micro-benchmarks on the DLAs to collect the performance data.
Revised: The data collection takes about two weeks for running 100,000 different samples of the TensorFlow micro-benchmarks on the DLAs to collect the performance data.

Original: On the other hand, for the
Revised: On the other hand, for the

Original: TensorRT micro
Revised: TensorRT micro

Original: -benchmarks, more than two weeks were spent to optimize and proﬁle the 25, different conﬁgurations of the TensorRT programs.
Revised: -benchmarks, more than two weeks were spent to optimize and proﬁle the 25, different conﬁgurations of the TensorRT programs.

Original: It is interesting to note that the TensorRT experiments generate large optimized intermediate ﬁles, especially for the dense layer, where it requires more than 5TB of storage space to keep its parameters.
Revised: It is interesting to note that the TensorRT experiments generate large optimized intermediate ﬁles, especially for the dense layer, where it requires more than 5TB of storage space to keep its parameters.

Original: Due to the disk space limitation, we select 16,000 out of 25,000 samples to run and proﬁle their performance.
Revised: Due to the disk space limitation, we select 16,000 out of 25,000 samples to run and proﬁle their performance.

Original: For data extraction, our data processing tool ﬁlters out the outliers (data with extreme values) before feeding the proﬁled data for the model training.
Revised: For data extraction, our data processing tool ﬁlters out the outliers (data with extreme values) before feeding the proﬁled data for the model training.

Original: The total elapsed time of each layer is decomposed into the preprocessing time ( T pre ) , the execution time ( T exe ) , and the postprocessing time ( T post ), as mentioned in the previous section.
Revised: The total elapsed time of each layer is decomposed into the preprocessing time ( T pre ) , the execution time ( T exe ) , and the postprocessing time ( T post ), as mentioned in the previous section.

Original: In order to test the accuracy of our trained model, the collected samples are split into 80% of the samples as training datasets and 20% as testing datasets.
Revised: In order to test the accuracy of our trained model, the collected samples are split into 80% of the samples as training datasets and 20% as testing datasets.

Original: 5. D ATA T RANSFORMATION
Revised: 5. D ATA T RANSFORMATION

Original: Now, suppose we are given a training dataset D , which is comprising m observations and p features of X and written as D
Revised: Now, suppose we are given a training dataset D , which is comprising m observations and p features of X and written as D

Original: = { t i , x i 1 , x i 2 , ..., x ip } mi =1 , where t is a vector of observed values t i ( i = 1 , ..., m ) , and X could be seen as a matrix of row-vectors x
Revised: = { t i , x i 1 , x i 2 , ..., x ip } mi =1 , where t is a vector of observed values t i ( i = 1 , ..., m ) , and X could be seen as a matrix of row-vectors x

Original: i ( i = 1 , ..., m ) or of mdimensional column-vectors X j ( j = 1 , ..., p ) .
Revised: i ( i = 1 , ..., m ) or of mdimensional column-vectors X j ( j = 1 , ..., p ) .

Original: The coefﬁcients vector w keeps the weights of the model.
Revised: The coefﬁcients vector w keeps the weights of the model.

Original: The predicted value is denoted as y ( x , w ) , for any given model of weights w and the dataset x .
Revised: The predicted value is denoted as y ( x , w ) , for any given model of weights w and the dataset x .

Original: In order to improve the convergence efﬁciency and stability of the stochastic gradient descent (SGD) algorithm, the three types of data transformations are adopted in this work, including scalar multiplication, Z-scores transformation, and Box-Cox transformation.
Revised: In order to improve the convergence efﬁciency and stability of the stochastic gradient descent (SGD) algorithm, the three types of data transformations are adopted in this work, including scalar multiplication, Z-scores transformation, and Box-Cox transformation.

Original: Scalar multiplication is used to provide ﬁne-grained updates of the SGD procedure and scales each observed value t i .
Revised: Scalar multiplication is used to provide ﬁne-grained updates of the SGD procedure and scales each observed value t i .

Original: Z-scores transformation puts each data feature X j from different sources into the same scale to eliminate the prejudicial bias of the features values.
Revised: Z-scores transformation puts each data feature X j from different sources into the same scale to eliminate the prejudicial bias of the features values.

Original: Box-Cox transformation converts the values of the features
Revised: Box-Cox transformation converts the values of the features

Original: X j to standard normal random variables, which would further improve the effectiveness of Z-scores transformation.
Revised: X j to standard normal random variables, which would further improve the effectiveness of Z-scores transformation.

Original: Details of these data transformations are available in Appendixes B, C and D.
Revised: Details of these data transformations are available in Appendixes B, C and D.

Original: 5. L OSS F UNCTION
Revised: 5. L OSS F UNCTION

Original: As the observed vector t is with the positive-skew distribution and often contains some noises contributed by the measurement errors, we ﬁne-tune the loss function as mean absolute percentage logarithmic error (MAPLE) for the prediction model (Wang et al.), as shown in Equation 7.
Revised: As the observed vector t is with the positive-skew distribution and often contains some noises contributed by the measurement errors, we ﬁne-tune the loss function as mean absolute percentage logarithmic error (MAPLE) for the prediction model (Wang et al.), as shown in Equation 7.

Original: To cope with the situation of the skewed distribution, the logarithmic operations for the predicted values 1 + y ( x i , w ) and the observed values 1 + t i , and the division operation on the observed values in MAPLE are expected to enhance the accuracy of the small data, which occurs frequently.
Revised: To deal with the situation of the skewed distribution, the logarithmic operations for the predicted values 1 + y ( x i , w ) and the observed values 1 + t i , and the division operation on the observed values in MAPLE are expected to enhance the accuracy of the small data, which occurs frequently.

Original: On the other head, the absolute value of MAPLE helps increase the resistance against outliers that may unexpectedly appear in the measured data.
Revised: On the other head, the absolute value of MAPLE helps increase the resistance against outliers that may unexpectedly appear in the measured data.

Original: Moreover, to prevent over-ﬁtting, L2 regularization is added to the loss function, where λ 2 is a scaling factor for the regularization.
Revised: Moreover, to prevent over-ﬁtting, L2 regularization is added to the loss function, where λ 2 is a scaling factor for the regularization.

Original: [Equation]
Revised: [Equation]

Original:  E VALUATION
Revised:  E VALUATION

Original: The layer-wise and model-wise performance results are evaluated to demonstrate the effectiveness of ResPerfNet in this section.
Revised: The layer-wise and model-wise performance results are evaluated to demonstrate the effectiveness of ResPerfNet in this section.

Original: In particular, we compare the layer-wise estimated execution time produced by ResPerfNet and the previous works to show that ResPerfNet is superior to other regression based approaches, such as polynomial regression, support vector regression and PerfNet.
Revised: In particular, we compare the layer-wise estimated execution time produced by ResPerfNet and the previous works to show that ResPerfNet is superior to other regression based approaches, such as polynomial regression, support vector regression and PerfNet.

Original: Three statistical metrics, including mean absolute percentage error (MAPE), root mean squared error (RMSE) and mean absolute error (MAE), are used to quantify the effectiveness for each tested performance modeling approach.
Revised: Three statistical metrics, including mean absolute percentage error (MAPE), root mean squared error (RMSE) and mean absolute error (MAE), are used to quantify the effectiveness for each tested performance modeling approach.

Original: In addition, to demonstrate the capability of ResPerfNet for the full model prediction, three popular CNNs are considered in the model-wise experiments, e.g., LeNet,
Revised: In addition, to demonstrate the capability of ResPerfNet for the full model prediction, three popular CNNs are considered in the model-wise experiments, e.g., LeNet,

Original: AlexNet, and VGG16.
Revised: AlexNet, and VGG16.

Original: Note that three data transformations mentioned in Section 5.3 are applied in
Revised: Note that three data transformations mentioned in Section 5.3 are applied in

Original: ResPerfNet by default unless speciﬁed otherwise.
Revised: ResPerfNet by default unless speciﬁed otherwise.

Original: The details of our experimental environments are listed in Appendix F.
Revised: The details of our experimental environments are listed in Appendix F.

Original: 6. L AYER - WISE E XECUTION T IME P REDICTION
Revised: 6. L AYER - WISE E XECUTION T IME P REDICTION

Original: Table 1 compares the MAPEs of the execution time for the convolutional layers, estimated by ResPerfNet and the prior works.
Revised: Table 1 compares the MAPEs of the execution time for the convolutional layers, estimated by ResPerfNet and the prior works.

Original: While appropriate parameter adjustments are applied to obtain parame- ters for better results, the MAPEs of polynomial regression, support vector regression, and XGBoost are over 29%, which means the error is quite large and indicates that the corresponding approaches are not capable of doing good performance prediction for the real applications.
Revised: While appropriate parameter adjustments are applied to obtain parame- ters for better results, the MAPEs of polynomial regression, support vector regression, and XGBoost are over 29%, which means the error is quite large and indicates that the corresponding approaches are not capable of doing good performance prediction for the real applications.

Original: On the contrary, the DL-based approaches, PerfNet and ResPerfNet, give more accurate estimations, which have less than 15% of the MAPEs.
Revised: On the contrary, the DL-based approaches, PerfNet and ResPerfNet, give more accurate estimations, which have less than 15% of the MAPEs.

Original: In particular, ResPerfNet outperforms the other approaches and has 11.75% and 14.23% of the MAPEs for the TensorFlow and TensorRT models.
Revised: In particular, ResPerfNet outperforms the other approaches and has 11.75% and 14.23% of the MAPEs for the TensorFlow and TensorRT models.

Original: The results suggest that ResPerfNet correctly associates the program characteristics to the performance model.
Revised: The results suggest that ResPerfNet correctly associates the program characteristics to the performance model.

Original: [Table]
Revised: [Table]

Original: To further look into the effectiveness of PerfNet and ResPerfNet and the impact of the Box-Cox transformation on the predicted results, Figure 2 plots the error curves of the TensorFlow convolutional layer using the PerfNet and ResPerfNet with and without performing the data transformation.
Revised: To further look into the effectiveness of PerfNet and ResPerfNet and the impact of the Box-Cox transformation on the predicted results, Figure 2 plots the error curves of the TensorFlow convolutional layer using the PerfNet and ResPerfNet with and without performing the data transformation.

Original: Figure 2(a) shows that most of the MAPE of ResPerfNet on the testing dataset are below 15%, as depicted by the red/black solid lines.
Revised: Figure 2(a) shows that most of the MAPE of ResPerfNet on the testing dataset are below 15%, as depicted by the red/black solid lines.

Original: Notably, ResPerfNet applying the Box-Cox data transformation reaches the lowest prediction error (11.7%), 2% less than ResPerfNet without the data transforming.
Revised: Notably, ResPerfNet applying the Box-Cox data transformation reaches the lowest prediction error (11.7%), 2% less than ResPerfNet without the data transforming.

Original: Similar trends can be observed in Figure 2(b) using the RMSE metric, in which the black solid line also shows the best performance.
Revised: Similar trends can be observed in Figure 2(b) using the RMSE metric, in which the black solid line also shows the best performance.

Original: The results presented in Figure 2 show that ResPerfNet with BoxCox transformation has better convergence rate, given the same training epoch.
Revised: The results presented in Figure 2 show that ResPerfNet with BoxCox transformation has better convergence rate, given the same training epoch.

Original: The detailed training process is illustrated in Appendix E. Moreover, the R 2 values for the predicted and measured execution time of the convolutional, pooling and dense layers are all above 0.97, which demonstrate high prediction quality of ResPerfNet, as illustrated in Figure 4 of Appendix I.
Revised: The detailed training process is illustrated in Appendix E. Moreover, the R 2 values for the predicted and measured execution time of the convolutional, pooling and dense layers are all above 0.97, which demonstrate high prediction quality of ResPerfNet, as illustrated in Figure 4 of Appendix I.

Original: The layer-wise performance results of the TensorFlow and TensorRT models delivered by ResPerfNet are listed in Table 2.
Revised: The layer-wise performance results of the TensorFlow and TensorRT models delivered by ResPerfNet are listed in Table 2.

Original: Overall, the MAPE for all phases are under 16%, which removes the concern of over-ﬁtting.
Revised: Overall, the MAPE for all phases are under 16%, which removes the concern of over-ﬁtting.

Original: For RMSE, the value of the TensorFlow version convolutional layer is 0.84ms.
Revised: For RMSE, the value of the TensorFlow version convolutional layer is 0.84ms.

Original: It is better than the 0.98ms reported by PerfNet (Wang et al.), and is also better than the 2.55ms produced by the method in (Justus et al.
Revised: It is better than the 0.98ms reported by PerfNet (Wang et al.), and is also better than the 2.55ms produced by the method in (Justus et al.

Original: Detailed predicted results of the three layers under different phases for TensorFlow (with 3 additional platforms) and for TensorRT are also presented in Appendixes G and H, respectively.
Revised: Detailed predicted results of the three layers under different phases for TensorFlow (with 3 additional platforms) and for TensorRT are also presented in Appendixes G and H, respectively.

Original: From the tables, we can see that ResPerfNet has better predicted results for TensorFlow than TensorRT.
Revised: From the tables, we can see that ResPerfNet has better predicted results for TensorFlow than TensorRT.

Original: That is because currently the TensorRTbased ResPerfNet is trained with less training data, as described in Section 5.2.
Revised: That is because currently the TensorRTbased ResPerfNet is trained with less training data, as described in Section 5.2.

Original: We believe that the accuracy for TensorRT predictions can be further improved with sufﬁcient data as TensorFlow.
Revised: We believe that the accuracy for TensorRT predictions can be further improved with sufﬁcient data as TensorFlow.

Original: [Table]
Revised: [Table]

Original: Figure 3 plots the inference time estimated by PerfNet and ResPerfNet for the three popular DNNs, including LeNet, AlexNet
Revised: Figure 3 plots the inference time estimated by PerfNet and ResPerfNet for the three popular DNNs, including LeNet, AlexNet

Original: [Table]
Revised: [Table]

Original: and VGG16, using TensorFlow and TensorRT frameworks.
Revised: and VGG16, using TensorFlow and TensorRT frameworks.

Original: Figure 3(a)- (c) shows that ResPerfNet has more accurate estimation than PerfNet since the averaged MAPE of the three models is 8.4% for all tested batch sizes, while PerfNet has the averaged MAPE of 24.04%.
Revised: Figure 3(a)- (c) shows that ResPerfNet has more accurate estimation than PerfNet since the averaged MAPE of the three models is 8.4% for all tested batch sizes, while PerfNet has the averaged MAPE of 24.04%.

Original: Figure 3(d)-(f) illustrates the similar trend for TensorRT based DNNs.
Revised: Figure 3(d)-(f) illustrates the similar trend for TensorRT based DNNs.

Original: The averaged MAPE of these
Revised: The averaged MAPE of these

Original: DNNs using ResPerfNet is 17%.
Revised: DNNs using ResPerfNet is 17%.

Original: The results show that our modeling and methodology are effective on the two popular frameworks
Revised: The results show that our modeling and methodology are effective on the two popular frameworks

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper
Revised: In this paper

Original: [Table]
Revised: [Table]

Original: we proposed a deep residual network architecture, ResPerfNet, to model the performance of neural networks on the target DLAs by considering the interactions between the host and the GPU and decomposing a neural network operation into three phases.
Revised: we proposed a deep residual network architecture, ResPerfNet, to model the performance of neural networks on the target DLAs by considering the interactions between the host and the GPU and decomposing a neural network operation into three phases.

Original: In addition, we apply ResPerfNet to predict the execution time of the optimized models, such as TensorRT, with the same performance characteristics as those used in unoptimized models.
Revised: In addition, we apply ResPerfNet to predict the execution time of the optimized models, such as TensorRT, with the same performance characteristics as those used in unoptimized models.

Original: Our experimental results show that ResPerfNet is able to provide high-accuracy estimations on various DLAs, which helps facilitate the exploration of proper neural network architectures built with various DL frameworks.
Revised: Our experimental results show that ResPerfNet is able to provide high-accuracy estimations on various DLAs, which helps facilitate the exploration of proper neural network architectures built with various DL frameworks.

Original: A F EATURES OF T RAINING D ATA
Revised: A F EATURES OF T RAINING D ATA

Original: [Table]
Revised: [Table]

Original: func
Revised: func

Original: [Table]
Revised: [Table]

Original: tion
Revised: tion

Original: [Table]
Revised: [Table]

Original: boolean number for utilizing an additional intercept on training input data
Revised: boolean number for utilizing an additional intercept on training input data

Original: [Table]
Revised: [Table]

Original: windows size factors for scaling down the input data.
Revised: windows size factors for scaling down the input data.

Original: [Table]
Revised: [Table]

Original: 1 In the scenario column, C, P
Revised: 1 In the scenario column, C, P

Original: [Table]
Revised: [Table]

Original: and D indicate which of the NN layer (i
Revised: and D indicate which of the NN layer (i

Original: [Table]
Revised: [Table]

Original: e., Convolutional, Pooling, and Dense layer)
Revised: e., Convolutional, Pooling, and Dense layer)

Original: the corresponding feature applies to.
Revised: the corresponding feature applies to.

Original: B S CALAR M ULTIPLICATION
Revised: B S CALAR M ULTIPLICATION

Original: Scalar multiplication is applied on the observed vector t as Equation 8 to magnify the prediction results since the original data are too small to provide accurate estimates.
Revised: Scalar multiplication is applied on the observed vector t as Equation 8 to magnify the prediction results since the original data are too small to provide accurate estimates.

Original: It is interesting to note that the scalar multiplication would be inefﬁcient for some commonly used loss function, such as
Revised: It is interesting to note that the scalar multiplication would be inefﬁcient for some commonly used loss function, such as

Original: [Table]
Revised: [Table]

Original: , based on our experiences; nevertheless, it works well with the MAPLE by making every gradient converging smoothly without frequently adjusting an appropriate learning rate in each epoch.
Revised: , based on our experiences; nevertheless, it works well with the MAPLE by making every gradient converging smoothly without frequently adjusting an appropriate learning rate in each epoch.

Original: [Equation]
Revised: [Equation]

Original: C Z- SCORES T RANSFORMATION
Revised: C Z- SCORES T RANSFORMATION

Original: Z-scores transformation is performed on each n-dimensional column-vector X j as Equation 9, where ¯
Revised: Z-scores transformation is performed on each n-dimensional column-vector X j as Equation 9, where ¯

Original: X j is the mean of the each column-vector X j , and σ j is the standard deviation of each column-vector
Revised: X j is the mean of the each column-vector X j , and σ j is the standard deviation of each column-vector

Original: X j .
Revised: X j .

Original: Z-scores transformation resales the values of the features to ensure the mean to be zero and the standard deviation to be one.
Revised: Z-scores transformation resales the values of the features to ensure the mean to be zero and the standard deviation to be one.

Original: The values of the features are rescaled within the range between zero and one, which is useful for gradient decent algorithms.
Revised: The values of the features are rescaled within the range between zero and one, which is useful for gradient decent algorithms.

Original: [Equation]
Revised: [Equation]

Original: D B OX -C OX T RANSFORM
Revised: D B OX -C OX T RANSFORM

Original: Box-Cox transformation transforms the input features X j into a normal distribution for the best model accuracy.
Revised: Box-Cox transformation transforms the input features X j into a normal distribution for the best model accuracy.

Original: Box-Cox transformation is shown in Equation 10, where λ 1 is the best approximation for the selected features.
Revised: Box-Cox transformation is shown in Equation 10, where λ 1 is the best approximation for the selected features.

Original: In our experiments, Box-Cox transformation is applied on Matrix Size and Kernel Size (See Table 3) for the convolutional layer data, and Matrix Size for the pooling layer data.
Revised: In our experiments, Box-Cox transformation is applied on Matrix Size and Kernel Size (See Table 3) for the convolutional layer data, and Matrix Size for the pooling layer data.

Original: [Equation]
Revised: [Equation]

Original: E T HE P ROPOSED G RADIENT D ESCENT A LGORITHM
Revised: E T HE P ROPOSED G RADIENT D ESCENT A LGORITHM

Original: Algorithm 1 is the pseudo-code of our proposed algorithm to train each phase of the layers.
Revised: Algorithm 1 is the pseudo-code of our proposed algorithm to train each phase of the layers.

Original: The required parameters are deﬁned as follows: optimizer : algorithm used to update the attributes of a neural network, lr scheduler : sets the learning rate of each parameter group to the initial lr times a given function, total epochs : total epochs of the neural network algorithm, lr : learning rate, bs : maximum of the batch size for each epoch, η : period of learning rate decay, γ : multiplicative factor of learning rate decay, and λ 2 : multiplicative factor for the weight penalty.
Revised: The required parameters are deﬁned as follows: optimizer : algorithm used to update the attributes of a neural network, lr scheduler : sets the learning rate of each parameter group to the initial lr times a given function, total epochs : total epochs of the neural network algorithm, lr : learning rate, bs : maximum of the batch size for each epoch, η : period of learning rate decay, γ : multiplicative factor of learning rate decay, and λ 2 : multiplicative factor for the weight penalty.

Original: Algorithm 

Revised: Algorithm

Original: The stochastic gradient descent algorithm proposed by ResperfNet, where our default settings for the our DL regression problems are optimizer =
Revised: The stochastic gradient descent algorithm proposed by ResperfNet, where our default settings for the our DL regression problems are optimizer =

Original: Adam, total epochs = 200 , lr = 0 .
Revised: Adam, total epochs = 200 , lr = 0 .

Original: 1 , bs = 128 , η = 40 , γ = 0 .
Revised: 1 , bs = 128 , η = 40 , γ = 0 .

Original: 5 , λ 2 = 0 .
Revised: 5 , λ 2 = 0 .

Original: 1 , and scaler = 10 .
Revised: 1 , and scaler = 10 .

Original: Require: α : Multiplicative factor for the weight, n : Current batch size, τ : Current iteration.
Revised: Require: α : Multiplicative factor for the weight, n : Current batch size, τ : Current iteration.

Original: 1: t ← scalar multiplication ( t , scalar )
Revised: 1: t ← scalar multiplication ( t , scalar )

Original: [Equation]
Revised: [Equation]

Original: Update t by Equation 8.
Revised: Update t by Equation 8.

Original: 2: x ← Z - scores ( Box
Revised: 2: x ← Z - scores ( Box

Original: Cox ( x ))
Revised: Cox ( x ))

Original: (cid:46) Update x by Equation 10 and 9.
Revised: (cid:46) Update x by Equation 10 and 9.

Original: 3: for e in total epochs do 4: lr ← lr scheduler (
Revised: 3: for e in total epochs do 4: lr ← lr scheduler (

Original: , γ ) (cid:46) Update the learning rate by scheduler.
Revised: , γ ) (cid:46) Update the learning rate by scheduler.

Original: 5: for b in ( m/bs + 1) do 6: α ← optimizer ( lr ) (cid:46) Update the weight factor by optimizer.
Revised: 5: for b in ( m/bs + 1) do 6: α ← optimizer ( lr ) (cid:46) Update the weight factor by optimizer.

Original: 7: n ← x [ b ∗ bs : min (( b + 1) ∗ bs, m )]
Revised: 7: n ← x [ b ∗ bs : min (( b + 1) ∗ bs, m )]

Original: (cid:46)
Revised: (cid:46)

Original: Calculate n (current batch size).
Revised: Calculate n (current batch size).

Original: 8: ∇ E n ← calculate gradient of E n on model w τ (cid:46)
Revised: 8: ∇ E n ← calculate gradient of E n on model w τ (cid:46)

Original: Calculate E n by Equation 7.
Revised: Calculate E n by Equation 7.

Original: 9: w τ +1 ← w τ − α ∇ E n ( w τ ) 10: τ ← τ +11: end for 12: end for
Revised: 9: w τ +1 ← w τ − α ∇ E n ( w τ ) 10: τ ← τ +11: end for 12: end for

Original: F E XPERIMENTAL S ETUP
Revised: F E XPERIMENTAL S ETUP

Original: The experiments are done on the Intel i7 processors with a variety of hardware accelerators listed in
Revised: The experiments are done on the Intel i7 processors with a variety of hardware accelerators listed in

Original: Table 4.
Revised: Table 4.

Original: TensorFlow 1.13.1 and TensorRT 5.0.2.6 with Python 3.6 are used to build the DL models, running on Ubuntu 18.04.4 LTS (kernel version 5.4.0-42-generic).
Revised: TensorFlow 1.13.1 and TensorRT 5.0.2.6 with Python 3.6 are used to build the DL models, running on Ubuntu 18.04.4 LTS (kernel version 5.4.0-42-generic).

Original: [Table]
Revised: [Table]

Original: G L AYER - WISE E XECUTION T IME P REDICTION FOR T ENSOR F LOW
Revised: G L AYER - WISE E XECUTION T IME P REDICTION FOR T ENSOR F LOW

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: H L AYER - WISE E XECUTION T IME P REDICTION FOR T ENSOR RT
Revised: H L AYER - WISE E XECUTION T IME P REDICTION FOR T ENSOR RT

Original: [Table]
Revised: [Table]

Original: I P REDICTED VS . M EASURED T IME (T ENSOR F LOW )
Revised: I P REDICTED VS . M EASURED T IME (T ENSOR F LOW )

Original: J M ODEL - WISE E XECUTION T IME P REDICTION FOR T ENSOR F LOW
Revised: J M ODEL - WISE E XECUTION T IME P REDICTION FOR T ENSOR F LOW