Original: S IMPLE A UGMENTATION G OES A L ONG W AY : ADRL FOR DNN Q UANTIZATION
Revised: S IMPLE A UGMENTATION G OES A L ONG W AY : ADRL FOR DNN Q UANTIZATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Mixed precision quantization improves DNN performance by assigning different layers with different bit-width values.
Revised: Mixed precision quantization improves DNN performance by assigning different layers with different bit-width values.

Original: Searching for the optimal bit-width for each layer, however, remains a challenge.
Revised: Searching for the optimal bit-width for each layer, however, remains a challenge.

Original: Deep Reinforcement Learning (DRL) shows some recent promise.
Revised: Deep Reinforcement Learning (DRL) shows some recent promise.

Original: It however suffers instability due to function approximation errors, causing large variances in the early training stages, slow convergence, and suboptimal policies in the mixed precision quantization problem.
Revised: It however suffers instability due to function approximation errors, causing large variances in the early training stages, slow convergence, and suboptimal policies in the mixed precision quantization problem.

Original: This paper proposes augmented DRL (ADRL) as a way to alleviate these issues.
Revised: This paper proposes augmented DRL (ADRL) as a way to alleviate these issues.

Original: This new strategy augments the neural networks in DRL with a complementary scheme to boost the performance of learning.
Revised: This new strategy augments the neural networks in DRL with a complementary scheme to boost the performance of learning.

Original: The paper examines the effectiveness of
Revised: The paper examines the effectiveness of

Original: ADRL both analytically and empirically, showing that it can produce more accurate quantized models than the state of the art DRL-based quantization while improving the learning speed by 4.5-64 × .
Revised: ADRL both analytically and empirically, showing that it can produce more accurate quantized models than the state of the art DRL-based quantization while improving the learning speed by 4.5-64 × .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: By reducing the number of bits needed to represent a model parameter of Deep Neural Networks (DNN), quantization (Lin et al., 2016; Park et al., 2017; Han et al., 2015; Zhou et al., 2018; Zhu et al., 2016; Hwang & Sung, 2014; Wu et al., 2016; Zhang et al., 2018; Köster et al., 2017; Ullrich et al., 2017; Hou & Kwok, 2018; Jacob et al., 2018) is an important way to reduce the size and improve the energy efﬁciency and speed of DNN.
Revised: By reducing the number of bits needed to represent a model parameter of Deep Neural Networks (DNN), quantization (Lin et al., 2016; Park et al., 2017; Han et al., 2015; Zhou et al., 2018; Zhu et al., 2016; Hwang & Sung, 2014; Wu et al., 2016; Zhang et al., 2018; Köster et al., 2017; Ullrich et al., 2017; Hou & Kwok, 2018; Jacob et al., 2018) is an important way to reduce the size and improve the energy efﬁciency and speed of DNN.

Original: Mixed precision quantization selects a proper bit-width for each layer of a DNN, offering more ﬂexibility than ﬁxed precision quantization.
Revised: Mixed precision quantization selects a proper bit-width for each layer of a DNN, offering more ﬂexibility than ﬁxed precision quantization.

Original: A major challenge to mixed precision quantization (Micikevicius et al., 2017; Cheng et al., 2018) is the conﬁguration search problem, that is, how to ﬁnd the appropriate bit-width for each DNN layer efﬁciently.
Revised: A major challenge to mixed precision quantization (Micikevicius et al., 2017; Cheng et al., 2018) is the conﬁguration search problem, that is, how to ﬁnd the appropriate bit-width for each DNN layer efﬁciently.

Original: The search space grows exponentially as the number of layers increases, and assessing each candidate conﬁguration requires a long time of training and evaluation of the DNN.
Revised: The search space grows exponentially as the number of layers increases, and assessing each candidate conﬁguration requires a long time of training and evaluation of the DNN.

Original: Research efforts have been drawn to mitigate the issue for help better tap into the power of mixed precision quantization.
Revised: Research efforts have been drawn to mitigate the issue for help better tap into the power of mixed precision quantization.

Original: Prior methods mainly fall into two categories: (i) automatic methods, such as reinforcement learning (RL) (Lou et al., 2019;
Revised: Prior methods mainly fall into two categories: (i) automatic methods, such as reinforcement learning (RL) (Lou et al., 2019;

Original: Gong et al., 2019; Wang et al., 2018; Yazdanbakhsh et al., 2018; Cai et al., 2020) and neural architecture search (NAS) (Wu et al., 2018; Li et al., 2020), to learn from feedback signals and automatically determine the quantization conﬁgurations; (ii) heuristic methods to reduce the search space under the guidance of metrics such as weight loss or Hessian spectrum (Dong et al., 2019; Wu et al., 2018;
Revised: Gong et al., 2019; Wang et al., 2018; Yazdanbakhsh et al., 2018; Cai et al., 2020) and neural architecture search (NAS) (Wu et al., 2018; Li et al., 2020), to learn from feedback signals and automatically determine the quantization conﬁgurations; (ii) heuristic methods to reduce the search space under the guidance of metrics such as weight loss or Hessian spectrum (Dong et al., 2019; Wu et al., 2018;

Original: Zhou et al., 2018; Park et al., 2017) of each layer.
Revised: Zhou et al., 2018; Park et al., 2017) of each layer.

Original: Comparing to the heuristic method, automatic methods, especially Deep Reinforcement Learning (DRL), require little human effort and give the state-of-the-art performance (e.g., via actor-critic setting (Whiteson et al., 2011; Zhang et al., 2016;
Revised: Comparing to the heuristic method, automatic methods, especially Deep Reinforcement Learning (DRL), require little human effort and give the state-of-the-art performance (e.g., via actor-critic set- Now ting (Whiteson et al., 2011; Zhang et al., 2016;

Original: Henderson et al., 2018; Wang et al., 2018)).
Revised: Henderson et al., 2018; Wang et al., 2018)).

Original: It however suffers from overestimation bias, high variance of the estimated value, and hence slow convergence and suboptimal results.
Revised: It however suffers from overestimation bias, high variance of the estimated value, and hence slow convergence and suboptimal results.

Original: The problem is fundamentally due to the poor function approximations given out by the DRL agent, especially during the early stage of the DRL learning process (Thrun & Schwartz, 1993; Anschel et al., 2017; Fujimoto et al., 2018) when the neural networks used in the DRL are of low quality.
Revised: The problem is fundamentally due to the poor function approximations given out by the DRL agent, especially during the early stage of the DRL learning process (Thrun & Schwartz, 1993; Anschel et al., 2017; Fujimoto et al., 2018) when the neural networks used in the DRL are of low quality.

Original: The issue prevents DRL from serving as a scalable solution to DNN quantization as DNN becomes deeper and more complex.
Revised: The issue prevents DRL from serving as a scalable solution to DNN quantization as DNN becomes deeper and more complex.

Original: This paper reports that simple augmentations can bring some surprising improvements to DRL for
Revised: This paper reports that simple augmentations can bring some surprising improvements to DRL for

Original: DNN quantization.
Revised: DNN quantization.

Original: We introduce augmented DRL (ADRL) as a principled way to signiﬁcantly magnify the potential of DRL for DNN quantization.
Revised: We introduce augmented DRL (ADRL) as a principled way to signiﬁcantly magnify the potential of DRL for DNN quantization.

Original: The principle of ADRL is to augment the neural networks in DRL with a complementary scheme (called augmentation scheme ) to complement the weakness of DRL policy approximator.
Revised: The principle of ADRL is to augment the neural networks in DRL with a complementary scheme (called augmentation scheme ) to complement the weakness of DRL policy approximator.

Original: Analytically, we prove the effects of such a method in reducing the variance and improving the convergence rates of DRL.
Revised: Analytically, we prove the effects of such a method in reducing the variance and improving the convergence rates of DRL.

Original: Empirically, we exemplify
Revised: Empirically, we exemplify

Original: ADRL with two example augmentation schemes and test on four popular DNNs.
Revised: ADRL with two example augmentation schemes and test on four popular DNNs.

Original: Comparisons with four prior DRL methods show that ADRL can shorten the quantization process by 4.5-64 × while improving the model accuracy substantially.
Revised: Comparisons with four prior DRL methods show that ADRL can shorten the quantization process by 4.5-64 × while improving the model accuracy substantially.

Original: It is worth mentioning that there is some prior work trying to increase the scalability of DRL.
Revised: It is worth mentioning that there is some prior work trying to increase the scalability of DRL.

Original: Dulac-Arnold et al.
Revised: Dulac-Arnold et al.

Original: (2016), for instance, addresses large discrete action spaces by embedding them into continuous spaces and leverages nearest neighbor to ﬁnd closest actions.
Revised: (2016), for instance, addresses large discrete action spaces by embedding them into continuous spaces and leverages nearest neighbor to ﬁnd closest actions.

Original: Our focus is different, aiming to enhance the learning speed of DRL by augmenting the weak policy approximator with complementary schemes.
Revised: Our focus is different, aiming to enhance the learning speed of DRL by augmenting the weak policy approximator with complementary schemes.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Deep Deterministic Policy Gradient (DDPG)
Revised: Deep Deterministic Policy Gradient (DDPG)

Original: A standard reinforcement learning system consists of an agent interacting with an environment E .
Revised: A standard reinforcement learning system consists of an agent interacting with an environment E .

Original: At each time step t , the agent receives an observation x t , takes an action a t and then receives an award r t .
Revised: At each time step t , the agent receives an observation x t , takes an action a t and then receives an award r t .

Original: Modeled with Markov decision process (MDP) with a state space S and an action space A , an agent’s behavior is deﬁned by a policy π : S
Revised: Modeled with Markov decision process (MDP) with a state space S and an action space A , an agent’s behavior is deﬁned by a policy π : S

Original: → A .
Revised: → A .

Original: A state is deﬁned as a sequence of actions and observations s t =
Revised: A state is deﬁned as a sequence of actions and observations s t =

Original: ( x 1 , a 2 , · · · , a t − 1 , x t ) when the environment is (partially) observed.
Revised: ( x 1 , a 2 , · · · , a t − 1 , x t ) when the environment is (partially) observed.

Original: For DNN quantization, the environment is assumed to be fully observable ( s t = x t ).
Revised: For DNN quantization, the environment is assumed to be fully observable ( s t = x t ).

Original: The return from a state s at time t is deﬁned as the future discounted return R t =
Revised: The return from a state s at time t is deﬁned as the future discounted return R t =

Original: (cid:80)
Revised: (cid:80)

Original: Ti = t γ i − t r
Revised: Ti = t γ i − t r

Original: ( s i , a i ) with a discount factor γ .
Revised: ( s i , a i ) with a discount factor γ .

Original: The goal of the agent is to learn a policy that maximizes the expected return from the start state J ( π ) =
Revised: The goal of the agent is to learn a policy that maximizes the expected return from the start state J ( π ) =

Original: E [ R 1 | π ] .
Revised: E [ R 1 | π ] .

Original: An RL agent in continuous action spaces can be trained through the actor-critic algorithm and the deep deterministic policy gradient (DDPG).
Revised: An RL agent in continuous action spaces can be trained through the actor-critic algorithm and the deep deterministic policy gradient (DDPG).

Original: The parameterized actor function µ ( s | θ µ ) speciﬁes the current policy and deterministically maps a state s to an action a .
Revised: The parameterized actor function µ ( s | θ µ ) speciﬁes the current policy and deterministically maps a state s to an action a .

Original: The critic network Q ( s, a ) is a neural network function for estimating the action-value E
Revised: The critic network Q ( s, a ) is a neural network function for estimating the action-value E

Original: [ R t | s t = s, a t = a, π ] ; it is parameterized with θ Q and is learned using the Bellman equation as Q-learning.
Revised: [ R t | s t = s, a t = a, π ] ; it is parameterized with θ Q and is learned using the Bellman equation as Q-learning.

Original: The critic is updated by minimizing the loss
Revised: The critic is updated by minimizing the loss

Original: L ( θ Q ) =
Revised: L ( θ Q ) =

Original: E [( y t − Q ( s t , a t | θ Q )) 2 ] , where y t = r
Revised: E [( y t − Q ( s t , a t | θ Q )) 2 ] , where y t = r

Original: ( s t , a 
Revised: ( s t , a t )

Original: [Equation] + [Equation]
Revised: + γQ ( s t +1 , µ ( s t +1 | θ µ )

Original: [Equation]
Revised: | θ Q ) .

Original: The actor is updated by applying the chain rule to the expected return J with respect to its parameters:
Revised: The actor is updated by applying the chain rule to the expected return J with respect to its parameters:

Original: [Equation]
Revised: [Equation]

Original: DDPG for Mixed Precision Quantization To apply the DRL to mixed precision quantization, previous work, represented by HAQ (Wang et al., 2018), uses DDPG as the agent learning policy.
Revised: DDPG for Mixed Precision Quantization To apply the DRL to mixed precision quantization, previous work, represented by HAQ (Wang et al., 2018), uses DDPG as the agent learning policy.

Original: The environment is assumed to be fully observed so that s t = x t , where the observation x t is deﬁned as x t
Revised: The environment is assumed to be fully observed so that s t = x t , where the observation x t is deﬁned as x t

Original: = ( l, c in , c out , s kernel , s stride , s feat , n params , i dw , i w/a , a t − 1 ) for convolution layers and x t =
Revised: = ( l, c in , c out , s kernel , s stride , s feat , n params , i dw , i w/a , a t − 1 ) for convolution layers and x t =

Original: ( l, h in , h out , 1 , 0 , s feat , n params , 0 , i w/a , a t − 1 ) for fully connected layers.
Revised: ( l, h in , h out , 1 , 0 , s feat , n params , 0 , i w/a , a t − 1 ) for fully connected layers.

Original: Here, l denotes the layer index, c in and c out are the numbers of input and output channels for the convolution layer, s kernel and s stride are the kernel size and stride for the convolution layer, h in and h out are the numbers of input and output hidden units for the fully connected layer, n params is the number of parameters, i dw
Revised: Here, l denotes the layer index, c in and c out are the numbers of input and output channels for the convolution layer, s kernel and s stride are the kernel size and stride for the convolution layer, h in and h out are the numbers of input and output hidden units for the fully connected layer, n params is the number of parameters, i dw

Original: and i w/a are binary indicators for depth-wise convolution and weight/activation, and a t − 1 is the action given by the agent from the previous step.
Revised: and i w/a are binary indicators for depth-wise convolution and weight/activation, and a t − 1 is the action given by the agent from the previous step.

Original: In time step t − 1 , the agent gives an action a t − 1 for layer l − 1 , leading to an observation x t .
Revised: In time step t − 1 , the agent gives an action a t − 1 for layer l − 1 , leading to an observation x t .

Original: Then the agent gives the action a t for layer l at time step t given x t .
Revised: Then the agent gives the action a t for layer l at time step t given x t .

Original: The agent updates the actor and critic networks after one episode following DDPG, which is a full pass of all the layers in the target neural network for quantization.
Revised: The agent updates the actor and critic networks after one episode following DDPG, which is a full pass of all the layers in the target neural network for quantization.

Original: The time step t and layer index l are interchangeable in this scenario.
Revised: The time step t and layer index l are interchangeable in this scenario.

Revised: [Figure]

Original: The systems use a continuous action space for precision prediction.
Revised: The systems use a continuous action space for precision prediction.

Original: At each time step t , the continuous action a t is mapped to the discrete bit value b k for layer k using b k = round ( b min − 0 .
Revised: At each time step t , the continuous action a t is mapped to the discrete bit value b k for layer k using b k = round ( b min − 0 .

Original: 5 + a k × ( b max − b min + 1)) .
Revised: 5 + a k × ( b max − b min + 1)) .

Original: The reward function is computed using R = λ
Revised: The reward function is computed using R = λ

Original: × ( acc quant − acc origin ) .
Revised: × ( acc quant − acc origin ) .

Original:  A UGMENTED D EEP R EINFORCEMENT L EARNING (ADRL)
Revised:  A UGMENTED D EEP R EINFORCEMENT L EARNING (ADRL)

Original: This section explains our proposal, ADRL.
Revised: This section explains our proposal, ADRL.

Original: It builds upon the default actor-critic algorithm and is trained with DDPG.
Revised: It builds upon the default actor-critic algorithm and is trained with DDPG.

Original: In the original actor-critic algorithm, the policy approximator is the actor networks, which generates one action and feeds it into the critic networks.
Revised: In the original actor-critic algorithm, the policy approximator is the actor networks, which generates one action and feeds it into the critic networks.

Original: The essence of ADRL is to[Figure] augment the policy approximator with a supplementary scheme.
Revised: The essence of ADRL is to augment the policy approximator with a supplementary scheme.

Original: The scheme may be in an arbitrary form, constructed with domain knowledge or other mechanisms.
Revised: The scheme may be in an arbitrary form, constructed with domain knowledge or other mechanisms.

Original: It should boost the weak policy approximator in accuracy especially when it is not yet well trained, and hence help efﬁciently reduce the variance and accelerate the convergence of the DRL algorithm.
Revised: It should boost the weak policy approximator in accuracy especially when it is not yet well trained, and hence help efﬁciently reduce the variance and accelerate the convergence of the DRL algorithm.

Original: 3. D ESIGN
Revised: 3. D ESIGN

Original: Components of ADRL Figure 1 illustrates the general ﬂow of ADRL that employs post augmentation.
Revised: Components of ADRL Figure 1 illustrates the general ﬂow of ADRL that employs post augmentation.

Original: The augmented policy approximator consists of two main components: an expanded actor network and a reﬁnement function.
Revised: The augmented policy approximator consists of two main components: an expanded actor network and a reﬁnement function.

Original: The expansion of the actor network makes it output multiple rather than one candidate action; the reﬁnement function derives the most promising candidate by processing those actions and feeds it to the critic networks.
Revised: The expansion of the actor network makes it output multiple rather than one candidate action; the reﬁnement function derives the most promising candidate by processing those actions and feeds it to the critic networks.

Original: The two components can be formally described as follows.
Revised: The two components can be formally described as follows.

Original:  generation The expanded actor function, ˆ µ θ ˆ µ : S → R k × n , maps a state from the state space R m to k ( k >1) actions in a continuous actions space R n : A k
Revised: Actions generation The expanded actor function, ˆ µ θ ˆ µ : S → R k × n , maps a state from the state space R m to k ( k >1) actions in a continuous actions space R n : A k

Original: = ˆ µ ( s | θ ˆ µ ) where A k =
Revised: = ˆ µ ( s | θ ˆ µ ) where A k =

Original: [ a T 1 , · · · , a Tk ] T .
Revised: [ a T 1 , · · · , a Tk ] T .

Original: The expansion can be done by modifying the last layer of the actor network (Sec 3.3).
Revised: The expansion can be done by modifying the last layer of the actor network (Sec 3.3).

Original: The outputs of this actor function serve as the candidate actions to the reﬁnement component.
Revised: The outputs of this actor function serve as the candidate actions to the reﬁnement component.

Original: Action reﬁnement.
Revised: Action reﬁnement.

Original: The action reﬁnement function, g : R k × n → R n , derives a promising action from the candidate actions.
Revised: The action reﬁnement function, g : R k × n → R n , derives a promising action from the candidate actions.

Original: A simple form of derivation is selection, that is, a ∗ =
Revised: A simple form of derivation is selection, that is, a ∗ =

Original: arg max
Revised: arg max

Original: a =
Revised: a =

Original: A k,i Q ( s , a )
Revised: A k,i Q ( s , a )

Original: Depending on how well the critic network is trained, the critic may not be able to consistently give a good estimation of Q ( s , a ) , especially at the early training stage.
Revised: Depending on how well the critic network is trained, the critic may not be able to consistently give a good estimation of Q ( s , a ) , especially at the early training stage.

Original: The augmented policy may use a Q-value indicator (cid:101) Q E ( a ) whose output depends on only the action and the property of the environment.
Revised: The augmented policy may use a Q-value indicator (cid:101) Q E ( a ) whose output depends on only the action and the property of the environment.

Original: Hence, g ( A k ) =
Revised: Hence, g ( A k ) =

Original: arg max a =
Revised: arg max a =

Original: A k,i (cid:101) Q E ( a )
Revised: A k,i (cid:101) Q E ( a )

Original: The choice of (cid:101) Q E ( a ) also depends on the speciﬁc tasks the ADRL is trying to solve.
Revised: The choice of (cid:101) Q E ( a ) also depends on the speciﬁc tasks the ADRL is trying to solve.

Original: Combining the generation function and the reﬁnement function, the augmented policy estimator can be represented as π θ ˆ µ ( s ) =
Revised: Combining the generation function and the reﬁnement function, the augmented policy estimator can be represented as π θ ˆ µ ( s ) =

Original: g ◦ ˆ µ θ ˆ µ .
Revised: g ◦ ˆ µ θ ˆ µ .

Original: Training with DDPG We train parameters for the actor and the critic networks using DDPG.
Revised: Training with DDPG We train parameters for the actor and the critic networks using DDPG.

Original: Although the augmented policy π θ ˆ µ consists of an actor network ˆ µ θ ˆ µ and a reﬁnement function g , the training of the full policy follows the policy gradient of ˆ µ θ ˆ µ because the effects of g are deterministic aspects of the environment E (Dulac-Arnold et al., 2015).
Revised: Although the augmented policy π θ ˆ µ consists of an actor network ˆ µ θ ˆ µ and a reﬁnement function g , the training of the full policy follows the policy gradient of ˆ µ θ ˆ µ because the effects of g are deterministic aspects of the environment E (Dulac-Arnold et al., 2015).

Original: The actor and the critic network are trained with variations of formulae (1) and (2):
Revised: The actor and the critic network are trained with variations of formulae (1) and (2):

Original: L ( θ Q ) =
Revised: L ( θ Q ) =

Original: E [( y t − Q ( s t , ˆ µ ( s t | θ ˆ µ ) | θ Q )) 2 ] , where y t = r
Revised: E [( y t − Q ( s t , ˆ µ ( s t | θ ˆ µ ) | θ Q )) 2 ] , where y t = r

Original: ( s t , π θ ˆ µ ( s t ))
Revised: ( s t , π θ ˆ µ ( s t ))

Original: + γQ ( s t +1 , ˆ µ ( s t +1 | θ ˆ µ ) | θ Q ) ∇ θ ˆ µ J ≈ E [ ∇ A k Q ( s, A k | θ Q ) | s = s t ,A k =ˆ µ ( s t ) · ∇ θ ˆ µ ˆ µ ( s | θ ˆ µ ) | s = s t ] .
Revised: + γQ ( s t +1 , ˆ µ ( s t +1 | θ ˆ µ ) | θ Q ) [Equation]

Original: 3. E FFECTS ON V ARIANCE AND L EARNING S PEED
Revised: 3. E FFECTS ON V ARIANCE AND L EARNING S PEED

Original: Variance Reduction
Revised: Variance Reduction

Original: The actor-critic algorithm for mixed-precision quantization is a combination of Q-learning and function approximation.
Revised: The actor-critic algorithm for mixed-precision quantization is a combination of Q-learning and function approximation.

Original: Therefore, it suffers from various types of function approximation errors which affect the training stability.
Revised: Therefore, it suffers from various types of function approximation errors which affect the training stability.

Original: Analysis below follows the terminology from previous work (Thrun & Schwartz, 1993).
Revised: Analysis below follows the terminology from previous work (Thrun & Schwartz, 1993).

Original: Deﬁnition 

Revised: Deﬁnition

Original: If actions derived by the augmented policy approximator leads to higher rewards comparing to actions given by the default actor network, the augmentation is an effective augmentation .
Revised: If actions derived by the augmented policy approximator leads to higher rewards comparing to actions given by the default actor network, the augmentation is an effective augmentation .

Original: Deﬁnition 2 Let Q ( s t , a t | θ Q ) be the value function of DRL at time step t , Q ∗ ( s t , a t ) = max
Revised: Deﬁnition 2 Let Q ( s t , a t | θ Q ) be the value function of DRL at time step t , Q ∗ ( s t , a t ) = max

Original: π Q π ( s t , a t ) be the optimal value function, and y ts,a = r ( s t , a t ) + γQ ( s t +1 , a t +1 | θ Q ) be the DRL target.
Revised: π Q π ( s t , a t ) be the optimal value function, and y ts,a = r ( s t , a t ) + γQ ( s t +1 , a t +1 | θ Q ) be the DRL target.

Original: Then the function approximation error denoted as δ t can be decomposed as ∆ t = Q ( s t , a t | θ Q ) − Q ∗ ( s t , a t )
Revised: Then the function approximation error denoted as δ t can be decomposed as ∆ t = Q ( s t , a t | θ Q ) − Q ∗ ( s t , a t )

Original: =
Revised: =

Original: Q ( s t , a t | θ Q ) − y ts,a + y ts,a − Q ∗ ( s t , a t )
Revised: Q ( s t , a t | θ Q ) − y ts,a + y ts,a − Q ∗ ( s t , a t )

Original: The ﬁrst two items form the Target Approximation Error (TAE), denoted as Z ts,a = Q ( s t , a t | θ Q ) − y ts,a .
Revised: The ﬁrst two items form the Target Approximation Error (TAE), denoted as Z ts,a = Q ( s t , a t | θ Q ) − y ts,a .

Original: Proposition 1 Assuming that TAE is a random process (Thrun & Schwartz, 1993) such that
Revised: Proposition 1 Assuming that TAE is a random process (Thrun & Schwartz, 1993) such that

Original: E [ Z ts,a ] = 0 , V ar [ Z ts,a ] = σ 2 s , and Cov [ Z t 1 s,a , Z t 2 s (cid:48) ,a (cid:48) ]
Revised: E [ Z ts,a ] = 0 , V ar [ Z ts,a ] = σ 2 s , and Cov [ Z t 1 s,a , Z t 2 s (cid:48) ,a (cid:48) ]

Original: = 0 for t 1 (cid:54) =
Revised: = 0 for t 1 (cid:54) =

Original: t 2 , then an effective augmentation leads to a smaller variance of the estimated Q value.
Revised: t 2 , then an effective augmentation leads to a smaller variance of the estimated Q value.

Original: Proof: The estimated Q value function can be expressed as Q ( s t , a t | θ Q ) =
Revised: Proof: The estimated Q value function can be expressed as Q ( s t , a t | θ Q ) =

Original: Z ts,a + r ( s t , a t ) + γQ ( s t +1 , a t +1 | θ Q ) =
Revised: Z ts,a + r ( s t , a t ) + γQ ( s t +1 , a t +1 | θ Q ) =

Original: Z ts,a + r ( s t , a t ) + γ [ Z t +1 s,a + r ( s t +1 , a t +1 ) + γQ ( s t +2 , a t +2 | θ Q )]
Revised: Z ts,a + r ( s t , a t ) + γ [ Z t +1 s,a + r ( s t +1 , a t +1 ) + γQ ( s t +2 , a t +2 | θ Q )]

Original: = (cid:80)
Revised: = (cid:80)

Original: Ti = t γ i − t
Revised: Ti = t γ i − t

Original: [ r ( s i , a i )
Revised: [ r ( s i , a i )

Original: + Z is,a ] .
Revised: + Z is,a ] .

Original: It is a function of rewards and TAEs.
Revised: It is a function of rewards and TAEs.

Original: Therefore, the variance of the estimated Q value will be proportional to the variance of the reward and TAE, such that
Revised: Therefore, the variance of the estimated Q value will be proportional to the variance of the reward and TAE, such that

Original: V ar
Revised: V ar

Original: [ Q ( s t , a t | θ Q )]
Revised: [ Q ( s t , a t | θ Q )]

Original: = (cid:80)
Revised: = (cid:80)

Original: Ti = t γ i − t
Revised: Ti = t γ i − t

Original: ( V ar [ r ( s i , a i )]
Revised: ( V ar [ r ( s i , a i )]

Original: + V ar
Revised: + V ar

Original: [ Z is,a ]) .
Revised: [ Z is,a ]) .

Original: Let [ r min , r max ] and [ (cid:101) r min , (cid:101) r max ] be the range of rewards associated with the default actor network and the augmented policy approximator respectively.
Revised: Let [ r min , r max ] and [ (cid:101) r min , (cid:101) r max ] be the range of rewards associated with the default actor network and the augmented policy approximator respectively.

Original: With an effective augmentation , we have r min < (cid:101) r min , and r max = (cid:101) r max .
Revised: With an effective augmentation , we have r min < (cid:101) r min , and r max = (cid:101) r max .

Original: It is easy to derive that V ar [ (cid:101) r ] < V ar [ r ] .
Revised: It is easy to derive that V ar [ (cid:101) r ] < V ar [ r ] .

Original: Since V ar [ Z is,a ] is the same for both of the cases, an effective augmentation leads to a smaller variance of the estimated Q value due to smaller reward variance.
Revised: Since V ar [ Z is,a ] is the same for both of the cases, an effective augmentation leads to a smaller variance of the estimated Q value due to smaller reward variance.

Original: (cid:3)
Revised: (cid:3)

Original: In ADRL, we use a Q-value indicator to help choose actions that tend to give higher quantization accuracy, leading to a higher reward.
Revised: In ADRL, we use a Q-value indicator to help choose actions that tend to give higher quantization accuracy, leading to a higher reward.

Original: Therefore, it helps reduce the variance of the estimated Q value, as conﬁrmed by the experiment results in Section 4.
Revised: Therefore, it helps reduce the variance of the estimated Q value, as conﬁrmed by the experiment results in Section 4.

Original: Effects on Learning Speed
Revised: Effects on Learning Speed

Original: Proposition 2 An effective augmentation leads to larger step size and hence faster updates of the critic network.
Revised: Proposition 2 An effective augmentation leads to larger step size and hence faster updates of the critic network.

Original: Proof: The critic network is trained by minimizing loss as shown in Eq.
Revised: Proof: The critic network is trained by minimizing loss as shown in Eq.

Original: Let δ t =
Revised: Let δ t =

Original: y t − Q ( s t , a t | θ Q ) .
Revised: y t − Q ( s t , a t | θ Q ) .

Original: By differentiating the loss function with respect to θ Q , we get the formula for updating the critic network as θ Q t +

Revised: By differentiating the loss function with respect to θ Q , we get the formula for updating the critic network as θ Q t +

Original: =
Revised: =

Original: θ Q t + α θ Q · δ t · ∇ θ Q L ( θ Q ) .
Revised: θ Q t + α θ Q · δ t · ∇ θ Q L ( θ Q ) .

Original: Let π θ ˆ µ be the augmented policy approximator and µ θ µ be the default actor network.
Revised: Let π θ ˆ µ be the augmented policy approximator and µ θ µ be the default actor network.

Original: With an effective augmentation , the augmented policy approximator should generate an action that leads to higher rewards such that r ( s t , π θ ˆ µ ( s t )) >
Revised: With an effective augmentation , the augmented policy approximator should generate an action that leads to higher rewards such that r ( s t , π θ ˆ µ ( s t )) >

Original: = r
Revised: = r

Original: ( s t , µ θ µ ( s t )) , and has a larger Q ( s t +1 , a t +1 ) .
Revised: ( s t , µ θ µ ( s t )) , and has a larger Q ( s t +1 , a t +1 ) .

Original: Assuming the same starting point for both π θ ˆ µ and µ θ µ , which means that Q ( s t , a t ) are the same for both of the two policies, we get δ π θ ˆ µ t > δ µ θµ t .
Revised: Assuming the same starting point for both π θ ˆ µ and µ θ µ , which means that Q ( s t , a t ) are the same for both of the two policies, we get δ π θ ˆ µ t > δ µ θµ t .

Original: Therefore, π θ ˆ µ leads to larger α θ Q · δ t · ∇ θ Q L ( θ Q ) , and thus a larger step size and hence faster updates to θ Q , the parameters in the critic network.
Revised: Therefore, π θ ˆ µ leads to larger α θ Q · δ t · ∇ θ Q L ( θ Q ) , and thus a larger step size and hence faster updates to θ Q , the parameters in the critic network.

Original: (cid:3)
Revised: (cid:3)

Original: Meanwhile, an effective augmentation helps DRL choose better-performing actions especially during the initial phases of the learning.
Revised: Meanwhile, an effective augmentation helps DRL choose better-performing actions especially during the initial phases of the learning.

Original: Together with the faster updates, it tends to make the learning converge faster, as consistently observed in our experiments reported in Section 4.
Revised: Together with the faster updates, it tends to make the learning converge faster, as consistently observed in our experiments reported in Section 4.

Original: 3. A PPLICATION ON M IXED -P RECISION Q UANTIZATION
Revised: 3. A PPLICATION ON M IXED -P RECISION Q UANTIZATION

Original: We now present how ADRL applies to mixed precision quantization.
Revised: We now present how ADRL applies to mixed precision quantization.

Original: The key question is how to design the augmented policy approximator for this speciﬁc task.
Revised: The key question is how to design the augmented policy approximator for this speciﬁc task.

Original: Our design takes the HAQ (Wang et al., 2018) as the base.
Revised: Our design takes the HAQ (Wang et al., 2018) as the base.

Original: For the actor networks, our expansion is to the last layer.
Revised: For the actor networks, our expansion is to the last layer.

Original: Instead of having only one neuron for the last layer, our action generator has k neurons corresponding to k candidate actions.
Revised: Instead of having only one neuron for the last layer, our action generator has k neurons corresponding to k candidate actions.

Original: We give a detailed explanation next.
Revised: We give a detailed explanation next.

Original: Q-value Indicator The augmentation centers around a Q-value indicator that we introduce to help select the most promising action from the candidates generated by the expanded actor function.
Revised: The augmentation centers around a Q-value indicator that we introduce to help select the most promising action from the candidates generated by the expanded actor function.

Original: To fulﬁll the requirement of the ADRL, the Q-value indicator should be independent from the learnable parameters of the actor and critic networks.
Revised: To fulﬁll the requirement of the ADRL, the Q-value indicator should be independent from the learnable parameters of the actor and critic networks.

Original: Its output shall be determined by the action a
Revised: Its output shall be determined by the action a

Original: and the environment E .
Revised: and the environment E .

Original: Since the Q-value is the expected return, it is directly connected with the inference accuracy after quantization ( acc quant ).
Revised: Since the Q-value is the expected return, it is directly connected with the inference accuracy after quantization ( acc quant ).

Original: In this work, we experiment with two forms of
Revised: In this work, we experiment with two forms of

Original: Q-value indicators.
Revised: Q-value indicators.

Original: They both estimate the relative ﬁtness of the candidate quantization levels in terms of the inﬂuence on acc quant .
Revised: They both estimate the relative ﬁtness of the candidate quantization levels in terms of the inﬂuence on acc quant .

Original: Their simplicities make them promising for actual adoptions.
Revised: Their simplicities make them promising for actual adoptions.

Original: Proﬁling-based indicator.
Revised: Proﬁling-based indicator.

Original: The ﬁrst form is to use the inference accuracy after quantization ( acc quant ) such that Q E ( a ) ∼ acc quant | quantize ( W l ,b ( a l )) .
Revised: The ﬁrst form is to use the inference accuracy after quantization ( acc quant ) such that Q E ( a ) ∼ acc quant | quantize ( W l ,b ( a l )) .

Original: At each time step t , the action generator generates k action candidates for layer l , where l = t .
Revised: At each time step t , the action generator generates k action candidates for layer l , where l = t .

Original: For each action a i =
Revised: For each action a i =

Original: A k,i , the Q-value indicator computes the corresponding bit value b i and quantizes layer l to b i bits while keeping all other layers unchanged.
Revised: A k,i , the Q-value indicator computes the corresponding bit value b i and quantizes layer l to b i bits while keeping all other layers unchanged.

Original: The quantized network is evaluated using a test dataset and the resulting accuracy is used as the indication of the ﬁtness of the quantization level.
Revised: The quantized network is evaluated using a test dataset and the resulting accuracy is used as the indication of the ﬁtness of the quantization level.

Original: The candidate giving the highest accuracy is selected by the action reﬁnement function g .
Revised: The candidate giving the highest accuracy is selected by the action reﬁnement function g .

Original: When there is a tie, the one that leads to a smaller network is selected by g .
Revised: When there is a tie, the one that leads to a smaller network is selected by g .

Original: Distance-based indicator.
Revised: Distance-based indicator.

Original: The second form is the distance between the original weight and the quantized weight given the bit value b ( a l ) for layer l .
Revised: The second form is the distance between the original weight and the quantized weight given the bit value b ( a l ) for layer l .

Original: We explore two forms of distance: L norm ( Q E ( a ) ∼ || W l − quantize ( W l , b ( a l ) || ) and KL-divergence (Wikipedia contributors, 2020) ( Q E ( a ) ∼ D KL ( W l || quantize ( W l , b ( a l ))) ).
Revised: We explore two forms of distance: L norm ( Q E ( a ) ∼ || W l − quantize ( W l , b ( a l ) || ) and KL-divergence (Wikipedia contributors, 2020) ( Q E ( a ) ∼ D KL ( W l || quantize ( W l , b ( a l ))) ).

Original: Both of them characterize the distance between the two weight distributions, and hence how sensitive a certain layer is with respect to a quantization level.
Revised: Both of them characterize the distance between the two weight distributions, and hence how sensitive a certain layer is with respect to a quantization level.

Original: As the distance depends only on the quantized bit value b l and the original weight, all the computations can be done ofﬂine ahead of time.
Revised: As the distance depends only on the quantized bit value b l and the original weight, all the computations can be done ofﬂine ahead of time.

Original: During the search, the reﬁnement function g selects the action that gives the smallest KL-divergence for the corresponding layer from the candidate pool.
Revised: During the search, the reﬁnement function g selects the action that gives the smallest KL-divergence for the corresponding layer from the candidate pool.

Original: Implementation Details
Revised: Implementation Details

Original: The entire process of mixed-precision quantization has three main stages: searching, ﬁnetuning, and evaluation.
Revised: The entire process of mixed-precision quantization has three main stages: searching, ﬁnetuning, and evaluation.

Original: The main focus of this work is the search stage, where we run the ADRL for several episodes to ﬁnd a quantization conﬁguration that satisﬁes our requirements (e.g., weight compression ratio).
Revised: The main focus of this work is the search stage, where we run the ADRL for several episodes to ﬁnd a quantization conﬁguration that satisﬁes our requirements (e.g., weight compression ratio).

Original: Then we ﬁnetune the model with the optimal conﬁguration for a number of epochs until the model converges.
Revised: Then we ﬁnetune the model with the optimal conﬁguration for a number of epochs until the model converges.

Original: A ﬁnal evaluation step gives the accuracy of the quantized model achieves.
Revised: A ﬁnal evaluation step gives the accuracy of the quantized model achieves.

Original: In each episode of the search stage, the agent generates a quantization conﬁguration with the guidance of both the quantized model accuracy and the model compression ratio.
Revised: In each episode of the search stage, the agent generates a quantization conﬁguration with the guidance of both the quantized model accuracy and the model compression ratio.

Original: First, the Q-value indicator selects the candidate network that gives the highest accuracy for each layer.
Revised: First, the Q-value indicator selects the candidate network that gives the highest accuracy for each layer.

Original: After selecting the bit values for all the layers, the agent checks whether the quantized network fulﬁlls the compression ratio requirement.
Revised: After selecting the bit values for all the layers, the agent checks whether the quantized network fulﬁlls the compression ratio requirement.

Original: If not, it will decrease the bit value layer by layer starting from the one that gives the least accuracy loss to generate a quantized model that fulﬁlls the compression ratio requirement.
Revised: If not, it will decrease the bit value layer by layer starting from the one that gives the least accuracy loss to generate a quantized model that fulﬁlls the compression ratio requirement.

Original: The accuracy of this quantized model is used to compute the reward and to train the agent.
Revised: The accuracy of this quantized model is used to compute the reward and to train the agent

Original: Memoization.
Revised: Memoization.

Original: When using the proﬁling based indicator, the agent needs to run inference on a subset of test dataset for each candidate action.
Revised: When using the proﬁling based indicator, the agent needs to run inference on a subset of test dataset for each candidate action.

Original: To save time, we introduce memoization into the process.
Revised: To save time, we introduce memoization into the process.

Original: We use a dictionary to store the inference accuracy given by different bit-width values for each layer.
Revised: We use a dictionary to store the inference accuracy given by different bit-width values for each layer.

Original: Given a candidate action a i for layer l , the agent maps it to a discrete bit value b
Revised: Given a candidate action a i for layer l , the agent maps it to a discrete bit value b

Original: i and checks if the inference accuracy of quantizing weights of layer l to b i bits is stored in the corresponding dictionary.
Revised: i and checks if the inference accuracy of quantizing weights of layer l to b i bits is stored in the corresponding dictionary.

Original: If so, the agent simply uses the stored value as the indicated Q-value for action a i .
Revised: If so, the agent simply uses the stored value as the indicated Q-value for action a i .

Original: Otherwise, it quantizes layer l to b i bits, evaluates the network to get the accuracy as the indicated Q-value, and adds the b i : acc quant ( b i ) pair into the dictionary.
Revised: Otherwise, it quantizes layer l to b i bits, evaluates the network to get the accuracy as the indicated Q-value, and adds the b i : acc quant ( b i ) pair into the dictionary.

Original: Early termination.
Revised: Early termination.

Original: A typical design for the searching stage runs the RL agent for a ﬁxed number of episodes and outputs the conﬁguration that gives the highest reward.
Revised: A typical design for the searching stage runs the RL agent for a ﬁxed number of episodes and outputs the conﬁguration that gives the highest reward.

Original: We observed that in the process of searching for optimal conﬁgurations, the time it takes for the acc quant to reach a stable status could be much less than the pre-set number of episodes.
Revised: We observed that in the process of searching for optimal conﬁgurations, the time it takes for the acc quant to reach a stable status could be much less than the pre-set number of episodes.

Original: Especially when using ADRL, the agent generates actions with high rewards at a very early stage.
Revised: Especially when using ADRL, the agent generates actions with high rewards at a very early stage.

Original: Therefore, we introduce an early termination strategy, which saves much computation time spending on the search process.
Revised: Therefore, we introduce an early termination strategy, which saves much computation time spending on the search process.

Original: The general idea for early termination is to stop the search process when the inference accuracy has small variance among consecutive runs.
Revised: The general idea for early termination is to stop the search process when the inference accuracy has small variance among consecutive runs.

Original: We use the standard derivation among N runs to measure the variance.
Revised: We use the standard derivation among N runs to measure the variance.

Original: If it is less than a threshold (detailed in Sec 4), the search process terminates.
Revised: If it is less than a threshold (detailed in Sec 4), the search process terminates.

Original: Time complexity.
Revised: Time complexity.

Original: Let T gen be the time of generating actions for each layer and T eval be the time spending on evaluating the quantized network once.
Revised: Let T gen be the time of generating actions for each layer and T eval be the time spending on evaluating the quantized network once.

Original: The time complexities of the original DRL method and ADRL without memorization are N · ( L · T gen + T eval ) and N (cid:48) · ( L · T gen +( kL +1) · T eval ) , where N and N (cid:48) are the number of search episodes for the original DRL method and ADRL, k is the number of candidate actions in ADRL for each layer and L is the number of layers of the target network.
Revised: The time complexities of the original DRL method and ADRL without memorization are N · ( L · T gen + T eval ) and N (cid:48) · ( L · T gen +( kL +1) · T eval ) , where N and N (cid:48) are the number of search episodes for the original DRL method and ADRL, k is the number of candidate actions in ADRL for each layer and L is the number of layers of the target network.

Original: The overhead brought by the Q-value indicator is N (cid:48)
Revised: The overhead brought by the Q-value indicator is N (cid:48)

Original: kL · T eval since it needs to evaluate the network accuracy for each candidate action in each layer.
Revised: kL · T eval since it needs to evaluate the network accuracy for each candidate action in each layer.

Original: With memorization , assuming n is the number of possible bit values in the search space, this overhead is largely reduced as the Q-value indicator only needs to evaluate the network n times for each layer during the entire search stage.
Revised: With memorization , assuming n is the number of possible bit values in the search space, this overhead is largely reduced as the Q-value indicator only needs to evaluate the network n times for each layer during the entire search stage.

Original: Therefore, if the network needs to be quantized to at most 8 bits, the overhead becomes 8 L · T eval .
Revised: Therefore, if the network needs to be quantized to at most 8 bits, the overhead becomes 8 L · T eval .

Original: The time complexity is hence reduced to N (cid:48) ·
Revised: The time complexity is hence reduced to N (cid:48) ·

Original: ( L · T gen + T eval ) + 8 L · T eval .
Revised: ( L · T gen + T eval ) + 8 L · T eval .

Original: With early termination,
Revised: With early termination,

Original: N (cid:48) is much smaller than N .
Revised: N (cid:48) is much smaller than N .

Original: Therefore, the total time spending on the search stage with ADRL is much smaller than that of the original DRL based method, which is demonstrated in the experiment results as shown in Sec 4.
Revised: Therefore, the total time spending on the search stage with ADRL is much smaller than that of the original DRL based method, which is demonstrated in the experiment results as shown in Sec 4.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: To evaluate the efﬁciency of augmented DRL on mixed precision quantization of DNNs, we experiment with four different networks: CifarNet (Krizhevsky, 2012), ResNet20 (He et al., 2016),
Revised: To evaluate the efﬁciency of augmented DRL on mixed precision quantization of DNNs, we experiment with four different networks: CifarNet (Krizhevsky, 2012), ResNet20 (He et al., 2016),

Original: AlexNet (Krizhevsky et al., 2012) and ResNet50 (He et al., 2016).
Revised: AlexNet (Krizhevsky et al., 2012) and ResNet50 (He et al., 2016).

Original: The ﬁrst two networks work on
Revised: The ﬁrst two networks work on

Original: Cifar10 (Krizhevsky, 2012) dataset while the last two work on the ImageNet (Deng et al., 2009) dataset.
Revised: Cifar10 (Krizhevsky, 2012) dataset while the last two work on the ImageNet (Deng et al., 2009) dataset.

Original: The experiments are done on a server with an
Revised: The experiments are done on a server with an

Original: Intel(R) Xeon(R)
Revised: Intel(R) Xeon(R)

Original: Platinum 8168 Processor, 32GB memory and 4 NVIDIA Tesla V100 32GB GPUs.
Revised: Platinum 8168 Processor, 32GB memory and 4 NVIDIA Tesla V100 32GB GPUs.

Original: The entire process of DRL based mixed-precision quantization consists of three main stages: search, ﬁnetuning, and evaluation.
Revised: The entire process of DRL based mixed-precision quantization consists of three main stages: search, ﬁnetuning, and evaluation.

Original: The search stage has E episodes.
Revised: The search stage has E episodes.

Original: Each episode consists of T time steps, where T equals the number of layers of the neural network.
Revised: Each episode consists of T time steps, where T equals the number of layers of the neural network.

Original: At each time step t , the RL algorithm selects a bit-width value for layer l = t .
Revised: At each time step t , the RL algorithm selects a bit-width value for layer l = t .

Original: After ﬁnishing 1 episode, it generates a mixed-precision conﬁguration with the bit-width values for all the layers, quantizes the network accordingly, and evaluates the quantized network to calculate the reward.
Revised: After ﬁnishing 1 episode, it generates a mixed-precision conﬁguration with the bit-width values for all the layers, quantizes the network accordingly, and evaluates the quantized network to calculate the reward.

Original: It then updates the actor and critic parameters and resets the target DNN network to the original non-quantized one.
Revised: It then updates the actor and critic parameters and resets the target DNN network to the original non-quantized one.

Original: The ﬁnetuning stage follows the search stage.
Revised: The ﬁnetuning stage follows the search stage.

Original: In this stage, we quantize the DNN network using the mixed-precision conﬁguration selected by the RL algorithm at the end of the search stage and ﬁnetune the network for N epochs.
Revised: In this stage, we quantize the DNN network using the mixed-precision conﬁguration selected by the RL algorithm at the end of the search stage and ﬁnetune the network for N epochs.

Original: The ﬁnal stage is evaluation, which runs the ﬁnetuned quantized model on the test dataset.
Revised: The ﬁnal stage is evaluation, which runs the ﬁnetuned quantized model on the test dataset.

Original: The resulting inference accuracy is the metric for the quality of the quantized network.
Revised: The resulting inference accuracy is the metric for the quality of the quantized network.

Original: We compare our ADRL with four prior mixed-precision quantization studies.
Revised: We compare our ADRL with four prior mixed-precision quantization studies.

Original: Besides the already introduced HAQ (Wang et al., 2018), the prior studies include ReLeQ (Yazdanbakhsh et al., 2018),
Revised: Besides the already introduced HAQ (Wang et al., 2018), the prior studies include ReLeQ (Yazdanbakhsh et al., 2018),

Original: HAWQ (Dong et al., 2019), and ZeroQ (Cai et al., 2020).
Revised: HAWQ (Dong et al., 2019), and ZeroQ (Cai et al., 2020).

Original: ReLeQ (Yazdanbakhsh et al., 2018) uses a unique reward formulation and shaping in its RL algorithm so that it can simultaneously optimize for two objectives (accuracy and reduced computation with lower bit width) with a uniﬁed RL process.
Revised: ReLeQ (Yazdanbakhsh et al., 2018) uses a unique reward formulation and shaping in its RL algorithm so that it can simultaneously optimize for two objectives (accuracy and reduced computation with lower bit width) with a uniﬁed RL process.

Original: HAWQ (Dong et al., 2019) automatically selects the relative quantization precision of each layer based on the layer’s Hessian spectrum and determines the ﬁne-tuning order for quantization layers based on second-order information.
Revised: HAWQ (Dong et al., 2019) automatically selects the relative quantization precision of each layer based on the layer’s Hessian spectrum and determines the ﬁne-tuning order for quantization layers based on second-order information.

Original: ZeroQ (Cai et al., 2020) enables mix-precision quantization without any access to the training or validation data.
Revised: ZeroQ (Cai et al., 2020) enables mix-precision quantization without any access to the training or validation data.

Original: It uses a Pareto frontier based method to automatically determine the mixed-precision bit setting for all layers with no manual search involved.
Revised: It uses a Pareto frontier based method to automatically determine the mixed-precision bit setting for all layers with no manual search involved.

Original: Our experiments use layer-wise quantization, but we note that ADRL can potentially augment policy approximators in quantizations at other levels (e.g. kernel-wise Lou et al.
Revised: Our experiments use layer-wise quantization, but we note that ADRL can potentially augment policy approximators in quantizations at other levels (e [Table] g. kernel-wise Lou et al.

Original: in a similar way.
Revised: in a similar way.

Original: For ADRL, we experimented with both P-ADRL (proﬁling-based) and D-ADRL (distance-based).
Revised: For ADRL, we experimented with both P-ADRL (proﬁling-based) and D-ADRL (distance-based).

Original: P-ADRL outperforms D-ADRL.
Revised: P-ADRL outperforms D-ADRL.

Original: We hence focus the overall results on P-ADRL, but offer the results of D-ADRL in the detailed analysis.
Revised: We hence focus the overall results on P-ADRL, but offer the results of D-ADRL in the detailed analysis.

Original: As ADRL is implemented based on HAQ, we collect detailed results on all experimented networks on HAQ (code downloaded from the original work (Wang et al., 2018)) to give a head-to-head comparison.
Revised: As ADRL is implemented based on HAQ [Table] we collect detailed results on all experimented networks on HAQ (code downloaded from the original work (Wang et al., 2018)) to give a head-to-head comparison.

Original: For the other three methods, we cite the numbers reported in the published papers.
Revised: For the other three methods, we cite the numbers reported in the published papers.

Original: The default setting of HAQ is searching for 600 episodes and ﬁnetuning for 100 epochs.
Revised: The default setting of HAQ is searching for 600 episodes and ﬁnetuning for 100 epochs.

Original: We also apply early termination to HAQ (HAQ+E), where, the termination criterion is that the coefﬁcient of variance of the averaged inference accuracy over 10 episodes is smaller than 0.01 for at least two consecutive 10-episode windows.
Revised: We also apply early termination to HAQ (HAQ+E), where, the termination criterion is that the coefﬁcient of variance of the averaged inference accuracy over 10 episodes is smaller than 0.01 for at least two consecutive 10-episode windows.

Original: In all experiments, k (number of candidate actions) is 3 for ADRL.
Revised: In all experiments, k (number of candidate actions) is 3 for ADRL.

Original: We report the quality of the quantization ﬁrst, and then focus on the speed and variance of the RL learning.
Revised: We report the quality of the quantization ﬁrst, and then focus on the speed and variance of the RL learning

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: Compression Ratios and Accuracy Table 1 compares the compression ratios and accuracy.
Revised: Compression Ratios and Accuracy Table 1 compares the compression ratios and accuracy.

Original: For the effects of the augmentation of the approximator, ADRL is the only method that delivers a comparable compression ratio without causing any accuracy loss on all three networks.
Revised: For the effects of the augmentation of the approximator, ADRL is the only method that delivers a comparable compression ratio without causing any accuracy loss on all three networks.

Original: Note that
Revised: Note that

Original: ADRL is more as a complement rather than a competitor to these work:
Revised: ADRL is more as a complement rather than a competitor to these work:

Original: The augmentation strategy it introduces can help complement the approximators in the existing DRL especially when they are not yet well trained.
Revised: The augmentation strategy it introduces can help complement the approximators in the existing DRL especially when they are not yet well trained.

Original: Learning Speed Table 2 reports the learning speed in each of the two major stages (search, ﬁnetune) and the overall learning process.
Revised: Learning Speed Table 2 reports the learning speed in each of the two major stages (search, ﬁnetune) and the overall learning process.

Original: The reported time is the total time before convergence.
Revised: The reported time is the total time before convergence.

Original: For the search stage, the time includes the overhead brought by the Q-value indicator.
Revised: For the search stage, the time includes the overhead brought by the Q-value indicator.

Original: As shown in columnand 8, the search stage is accelerated by 6–66 × , while the ﬁnetune stage by up to 20 × .
Revised: As shown in columnand 8, the search stage is accelerated by 6–66 × , while the ﬁnetune stage by up to 20 × .

Original: CifarNet is an exception.
Revised: CifarNet is an exception.

Original: For its simplicity, all methods converge in only one epoch in ﬁnetuning; because HAQ has a larger compression ratio as shown in Figure 2, neither P-ADRL nor HAQ+E has a speedup in ﬁnetuning over HAQ.
Revised: For its simplicity, all methods converge in only one epoch in ﬁnetuning; because HAQ has a larger compression ratio as shown in Figure 2, neither P-ADRL nor HAQ+E has a speedup in ﬁnetuning over HAQ.

Original: But they take much fewer episodes to search and hence still get large overall speedups.
Revised: But they take much fewer episodes to search and hence still get large overall speedups.

Original: Overall, P-ADRL shortens RDL learning by HAQ by a factor of 4.5 to 64.1, while HAQ+E gets only 1.2 to 8.8.
Revised: Overall, P-ADRL shortens RDL learning by HAQ by a factor of 4.5 to 64.1, while HAQ+E gets only 1.2 to 8.8.

Original: The reasons for the speedups come from the much reduced numbers of episodes needed for the DRL to converge in search [Table] and the higher accuracy (and hence earlier termination) that P-ADRL gives in the ﬁnetuning process.
Revised: The reasons for the speedups come from the much reduced numbers of episodes needed for the DRL to converge in search, and the higher accuracy (and hence earlier termination) that P-ADRL gives in the ﬁnetuning process.

Original: We next give a detailed analysis by looking into each of the two stages in further depth
Revised: We next give a detailed analysis by looking into each of the two stages in further depth.

Original: [Table] [Table] Search Stage Figure 2 shows the inference accuracy (acc), the coefﬁcient of variance (CV) of the inference accuracy, and the compression ratio in terms of the weight ratio (wratio) of the network for the search stage.
Revised: Search Stage Figure 2 shows the inference accuracy (acc), the coefﬁcient of variance (CV) of the inference accuracy, and the compression ratio in terms of the weight ratio (wratio) of the network for the search stage.

Original: The ﬁrst row of Figure 2 gives the inference accuracy results in the search stage.
Revised: The ﬁrst row of Figure 2 gives the inference accuracy results in the search stage.

Original: we[Figure] can see that the conﬁgurations found by P-ADRL give comparable inference accuracy for CifarNet and ResNet20 on Cifar10 dataset and better accuracy for AlexNet and ResNet50 on ImageNet dataset comparing to that of HAQ.
Revised: we can see that the conﬁgurations found by P-ADRL give comparable inference accuracy for CifarNet and ResNet20 on Cifar10 dataset and better accuracy for AlexNet and ResNet50 on ImageNet dataset comparing to that of HAQ.

Original: Also, with early termination, our strategy terminates with a much smaller number of search episodes comparing to HAQ.
Revised: Also, with early termination, our strategy terminates with a much smaller number of search episodes comparing to HAQ.

Original: As shown in column 3 of Table 2, the numbers of search episodes of P-ADRL are 19.1% - 34.5% of that of HAQ+E and 2.2% - 12.3% of that of the original HAQ.
Revised: As shown in column 3 of Table 2, the numbers of search episodes of P-ADRL are 19.1% - 34.5% of that of HAQ+E and 2.2% - 12.3% of that of the original HAQ.

Original: The performance of D-ADRL is between that of P-ADRL and HAQ on CifarNet,
Revised: The performance of D-ADRL is between that of P-ADRL and HAQ on CifarNet,

Original: ResNet20 and ResNet50 and the worst on AlexNet.
Revised: ResNet20 and ResNet50 and the worst on AlexNet.

Original: P-ADRL is more promise than D-ADRL does.
Revised: P-ADRL is more promise than D-ADRL does.

Revised: [Table]

Revised: [Figure]

Revised: The horizontal dashed light blue line is the accuracy of the original network before the quantization.

Revised: The blue and green lines are the accuracy result for our strategies (P-ADRL and D-ADRL) while the dashed red line shows the result for HAQ.

Revised: The two vertical lines mark the episodes at which the strategies terminated with early termination.

Original: The accuracy given by P-ADRL is much stabler than that of the HAQ.
Revised: The accuracy given by P-ADRL is much stabler than that of the HAQ.

Original: We use the coefﬁcient of variance (CV) to measure the stability of the inference accuracy.
Revised: We use the coefﬁcient of variance (CV) to measure the stability of the inference accuracy.

Original: As shown in the second row of
Revised: As shown in the second row of

Original: Figure 2, P-ADRL achieves a small CV much faster than HAQ on all four networks.
Revised: Figure 2, P-ADRL achieves a small CV much faster than HAQ on all four networks.

Original: This is the reason that P-ADRL terminates in fewer episodes than HAQ and HAQ+E do; this conclusion holds regardless of what variance threshold is used in the early termination as the CVs of P-ADRL are consistently smaller than those of HAQ.
Revised: This is the reason that P-ADRL terminates in fewer episodes than HAQ and HAQ+E do; this conclusion holds regardless of what variance threshold is used in the early termination as the CVs of P-ADRL are consistently smaller than those of HAQ.

Original: The third row of Figure 2 shows the compression ratio after quantization at each episode for each of the three algorithms.
Revised: The third row of Figure 2 shows the compression ratio after quantization at each episode for each of the three algorithms.

Original: The target ratio is set to 0.1.
Revised: The target ratio is set to 0.1.

Original: All three strategies fulﬁll the size limit requirement.
Revised: All three strategies fulﬁll the size limit requirement.

Original: By the end of the search stage, P-ADRL achieves a compression ratio similar to HAQ on large networks (ResNet20, AlexNet and ResNet50).
Revised: By the end of the search stage, P-ADRL achieves a compression ratio similar to HAQ on large networks (ResNet20, AlexNet and ResNet50).

Original: On a small network CifarNet, the compression ratio given by P-ADRL is slightly larger than that of HAQ, but still smaller than 0.1.
Revised: On a small network CifarNet, the compression ratio given by P-ADRL is slightly larger than that of HAQ, but still smaller than 0.1.

Original: Finetuning Stage After getting network precision from the search stage, we quantize the networks and ﬁnetune for 100 epochs.
Revised: Finetuning Stage After getting network precision from the search stage, we quantize the networks and ﬁnetune for 100 epochs.

Original: The learning rate is set as 0.001, which is the default setting from HAQ.
Revised: The learning rate is set as 0.001, which is the default setting from HAQ.

Original:  Figure 3 illustrates the accuracy results of the ﬁnetune stages for three different precision settings: the one selected by P-ADRL; the one selected by HAQ+E and the one selected by the original HAQ.
Revised: [Figure] Figure 3 illustrates the accuracy results of the ﬁnetune stages for three different precision settings: the one selected by P-ADRL; the one selected by HAQ+E and the one selected by the original HAQ.

Original: For CifarNet, all three achieve similar accuracies, higher than that of the original.
Revised: For CifarNet, all three achieve similar accuracies, higher than that of the original.

Original: For the other[Figure] three networks, P-ADRL gets the largest accuracy, fully recovering the original accuracy.
Revised: For the other three networks, P-ADRL gets the largest accuracy, fully recovering the original accuracy.

Original: HAQ+E performs the worst, suffering at least 1% accuracy loss.
Revised: HAQ+E performs the worst, suffering at least 1% accuracy loss.

Original: The sixth column of Table 2 gives the number of epochs each strategy needs for ﬁnetuning before reaching the accuracy of the original network; ’100’ means that the strategy cannot recover the original accuracy in the 100-epoch ﬁnetuning stage.
Revised: The sixth column of Table 2 gives the number of epochs each strategy needs for ﬁnetuning before reaching the accuracy of the original network; ’100’ means that the strategy cannot recover the original accuracy in the 100-epoch ﬁnetuning stage.

Original: P-ADRL takes 1–35 epochs to converge to the original accuracies, much fewer than HAQ and HAQ+E.
Revised: P-ADRL takes 1–35 epochs to converge to the original accuracies, much fewer than HAQ and HAQ+E.

Original: Finetuning hence stops earlier.
Revised: Finetuning hence stops earlier.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper has demonstrated that simple augmentation goes a long way in boosting DRL for DNN mixed precision quantization.
Revised: This paper has demonstrated that simple augmentation goes a long way in boosting DRL for DNN mixed precision quantization.

Original: It is easy to apply ADRL to deeper models such as ResNetand Resnet-152.
Revised: It is easy to apply ADRL to deeper models such as ResNetand Resnet-152.

Original: In principle, it is also possible to extend the method to latest model architectures as the basic ideas of the reinforcement learning and the Q-value indicator are independent of the detailed architecture of the network model.
Revised: In principle, it is also possible to extend the method to latest model architectures as the basic ideas of the reinforcement learning and the Q-value indicator are independent of the detailed architecture of the network model.

Original: Although the design of ADRL is motivated by DNN quantization, with appropriate augmentation schemes, the concept is potentially applicable to other
Revised: Although the design of ADRL is motivated by DNN quantization, with appropriate augmentation schemes, the concept is potentially applicable to other

Original: RL applications—a direction worth future exploration.
Revised: RL applications—a direction worth future exploration

Original: AA PPENDIX A. D ETAILS OF THE MIXED QUANTIZATION BITS FOR EACH MODEL
Revised: A PPENDIX A. D ETAILS OF THE MIXED QUANTIZATION BITS FOR EACH MODEL

Original: Figure 4 shows the comparison between the quantized networks selected using ADRL and HAQ.
Revised: Figure 4 shows the comparison between the quantized networks selected using ADRL and HAQ.

Original: In each plot, the upper half shows the bit value being used for each layer for the network quantized with
Revised: In each plot, the upper half shows the bit value being used for each layer for the network quantized with

Original: ADRL.
Revised: ADRL.

Original: The bottom half are the quantization conﬁgurations of the networks being quantized with
Revised: The bottom half are the quantization conﬁgurations of the networks being quantized with

Original: HAQ.
Revised: HAQ.

Original: [Figure]
Revised: [Figure]

Original: i t ( 
Revised: i t ( A

Revised: D R L

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: A. E VOLUTION OF THE SELECTED BIT VALUES
Revised: A. E VOLUTION OF THE SELECTED BIT VALUES

Original: [Figure]
Revised: [Figure]

Original: Figure 5 illustrates how the selected bit values change during the search stage for some layers of the four networks.
Revised: Figure 5 illustrates how the selected bit values change during the search stage for some layers of the four networks.

Original: The bit value being selected for thesame layer changes from one episode to the next.
Revised: The bit value being selected for the layer changes from one episode to the next.

Original: It spans well in the action search space.
Revised: It spans well in the action search space.