Original: A DAPTIVE C ORRELATED M ONTE C ARLO FOR C ONTEXTUAL C ATEGORICAL S EQUENCE G ENERATION
Revised: A DAPTIVE C ORRELATED M ONTE C ARLO FOR C ONTEXTUAL C ATEGORICAL S EQUENCE G ENERATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Sequence generation models are commonly reﬁned with reinforcement learning over user-deﬁned metrics.
Revised: Sequence generation models are commonly reﬁned with reinforcement learning over user-deﬁned metrics.

Original: However, high gradient variance hinders the practical use of this method.
Revised: However, high gradient variance hinders the practical use of this method.

Original: To stabilize this method, we adapt to contextual generation of categorical sequences a policy gradient estimator, which evaluates a set of correlated Monte Carlo (MC) rollouts for variance control.
Revised: To stabilize this method, we adapt to contextual generation of categorical sequences a policy gradient estimator, which evaluates a set of correlated Monte Carlo (MC) rollouts for variance control.

Original: Due to the correlation, the number of unique rollouts is random and adaptive to model uncertainty; those rollouts naturally become baselines for each other, and hence are combined to effectively reduce gradient variance.
Revised: Due to the correlation, the number of unique rollouts is random and adaptive to model uncertainty; those rollouts naturally become baselines for each other, and hence are combined to effectively reduce gradient variance.

Original: We also demonstrate the use of correlated MC rollouts for binary-tree softmax models which reduce the high generation cost in large vocabulary scenarios, by decomposing each categorical action into a sequence of binary actions.
Revised: We also demonstrate the use of correlated MC rollouts for binary-tree softmax models, which reduce the high generation cost in large vocabulary scenarios by decomposing each categorical action into a sequence of binary actions.

Original: We evaluate our methods on both neural program synthesis and image captioning.
Revised: We evaluate our methods on both neural program synthesis and image captioning.

Original: The proposed methods yield lower gradient variance and consistent improvement over related baselines.
Revised: The proposed methods yield lower gradient variance and consistent improvement over related baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Contextual categorical sequence generation is a core modeling component in a wide variety of machine learning tasks, such as neural program synthesis (Bunel et al., 2018;
Revised: Contextual categorical sequence generation is a core modeling component in a wide variety of machine learning tasks, such as neural program synthesis (Bunel et al., 2018;

Original: Devlin et al., 2017b;
Revised: Devlin et al., 2017b;

Original: Si et al., 2018; Chen et al., 2019) and image captioning (Vinyals et al., 2015; Xu et al., 2015).
Revised: Si et al., 2018; Chen et al., 2019) and image captioning (Vinyals et al., 2015; Xu et al., 2015).

Original: Typically, an encoder-decoder framework is applied.
Revised: Typically, an encoder-decoder framework is applied.

Original: The encoder maps a contextual input to a latent representation, conditioning on which and previously generated tokens the decoder generates categorical tokens in a consecutive manner (Bahdanau et al., 2014; Sutskever et al., 2014; Cho et al., 2014; Rush et al., 2015; Chopra et al., 2016).
Revised: The encoder maps a contextual input to a latent representation, conditioning on which and previously generated tokens the decoder generates categorical tokens in a consecutive manner (Bahdanau et al., 2014; Sutskever et al., 2014; Cho et al., 2014; Rush et al., 2015; Chopra et al., 2016).

Original: It is common to train contextual sequence generation models using maximum likelihood estimation (MLE), which attempts to maximize the likelihood of each token in a target sequence given its preceding tokens.
Revised: It is common to train contextual sequence generation models using maximum likelihood estimation (MLE), which attempts to maximize the likelihood of each token in a target sequence given its preceding tokens.

Original: Learning with MLE is often sub-optimal as it does not directly optimize the evaluation metric of the end task.
Revised: Learning with MLE is often sub-optimal as it does not directly optimize the evaluation metric of the end task.

Original: It generally suffers from the exposure bias (Bengio et al., 2015; Ranzato et al., 2016) , which refers to the discrepancy between training and generation using the Teacher Forcing (Williams & Zipser, 1989) strategy, i.e. , during training ground truth tokens are used as inputs, while during generation, only generated tokens are available.
Revised: It generally suffers from the exposure bias (Bengio et al., 2015; Ranzato et al., 2016) , which refers to the discrepancy between training and generation using the Teacher Forcing (Williams & Zipser, 1989) strategy, i.e. , during training ground truth tokens are used as inputs, while during generation, only generated tokens are available.

Original: Thus giving higher likelihoods to target sequences does not guarantee the model to generate sequences close to the target or good sequences.
Revised: Thus giving higher likelihoods to target sequences does not guarantee the model to generate sequences close to the target or good sequences.

Original: Moreover, MLE requires target sequences for training, while for many scenarios in task-oriented dialogue (Williams & Young, 2007) and program synthesis (Zhong et al., 2017), only the ﬁnal rewards to the generated sequences are available.
Revised: Moreover, MLE requires target sequences for training, while for many scenarios in task-oriented dialogue (Williams & Young, 2007) and program synthesis (Zhong et al., 2017), only the ﬁnal rewards to the generated sequences are available.

Original: To overcome the aforementioned issues of MLE, it is common to reﬁne a contextual sequence generation model pre-trained with MLE under the reinforcement learning (RL) framework (Zaremba & Sutskever, 2015; Ranzato et al., 2016; Bahdanau et al., 2016; Wu et al., 2018; Paulus et al., 2017).
Revised: To overcome the aforementioned issues of MLE, it is common to reﬁne a contextual sequence generation model pre-trained with MLE under the reinforcement learning (RL) framework (Zaremba & Sutskever, 2015; Ranzato et al., 2016; Bahdanau et al., 2016; Wu et al., 2018; Paulus et al., 2017).

Original: The objective becomes maximizing the expected rewards of model generated sequences.
Revised: The objective becomes maximizing the expected rewards of model generated sequences.

Original: During training, only the model generated tokens are fed into the model so that the exposure bias is avoided.
Revised: During training, only the model generated tokens are fed into the model so that the exposure bias is avoided.

Original: The reward to guide RL can be: 1) a task-dependent user-deﬁned metric, such as CIDEr for image captioning (Vedantam et al., 2015) and Generalization for neural program synthesis (Bunel et al., 2018); and 2) automatically learned reward using a discriminator or language model
Revised: The reward to guide RL can be: 1) a task-dependent user-deﬁned metric, such as CIDEr for image captioning (Vedantam et al., 2015) and Generalization for neural program synthesis (Bunel et al., 2018); and 2) automatically learned reward using a discriminator or language model

Original: (Yang et al., 2018; Yu et al., 2017; Lamb et al., 2016; Caccia et al., 2018; d’Autume et al., 2019).
Revised: (Yang et al., 2018; Yu et al., 2017; Lamb et al., 2016; Caccia et al., 2018; d’Autume et al., 2019).

Original: The RL training  enables direct improvement of the user-deﬁned or learned reward.
Revised: The RL training ICLR enables direct improvement of the user-deﬁned or learned reward.

Original: Moreover, in cases where only weak-supervision is available, e.g. , in neural program synthesis, RL may considerably improve the model performance (Bunel et al., 2018; Zhong et al., 2017).
Revised: Moreover, in cases where only weak-supervision is available, e.g. , in neural program synthesis, RL may considerably improve the model performance (Bunel et al., 2018; Zhong et al., 2017).

Original: However, the gradients of the expected reward in RL often suffer from high Monte Carlo (MC) estimation variance, due to noisy and/or sparse rewards and the large action space that grows exponentially with the sequence length.
Revised: However, the gradients of the expected reward in RL often suffer from high Monte Carlo (MC) estimation variance, due to noisy and/or sparse rewards and the large action space that grows exponentially with the sequence length.

Original: There has been signiﬁcant recent interest in variance reduction methods for MC gradient estimation (Mohamed et al., 2019).
Revised: There has been signiﬁcant recent interest in variance reduction methods for MC gradient estimation (Mohamed et al., 2019).

Original: A highly effective solution is the reparameterization trick (Kingma &
Revised: A highly effective solution is the reparameterization trick (Kingma &

Original: Welling, 2013; Rezende et al., 2014), which, however, is applicable to neither discrete variables nor non-differentiable reward functions.
Revised: Welling, 2013; Rezende et al., 2014), which, however, is applicable to neither discrete variables nor non-differentiable reward functions.

Original: For variance reduction involving discrete variables, one potential solution is to combine the Gumbel-softmax trick, which relaxes the discrete variables to continuous ones, with reparameterization to produce low-variance but biased gradients (Jang et al., 2017; Maddison et al., 2017).
Revised: For variance reduction involving discrete variables, one potential solution is to combine the Gumbel-softmax trick, which relaxes the discrete variables to continuous ones, with reparameterization to produce low-variance but biased gradients (Jang et al., 2017; Maddison et al., 2017).

Original: Another common way for variance reduction is adding appropriate baselines (Owen, 2013; Williams, 1992;
Revised: Another common way for variance reduction is adding appropriate baselines (Owen, 2013; Williams, 1992;

Original: Paisley et al., 2012; Ranganath et al., 2014; Mnih & Gregor, 2014), and there exist several such methods customized for discrete variables (Tucker et al., 2017;
Revised: Paisley et al., 2012; Ranganath et al., 2014; Mnih & Gregor, 2014), and there exist several such methods customized for discrete variables (Tucker et al., 2017;

Original: Grathwohl et al., 2018).
Revised: Grathwohl et al., 2018).

Original: However, due to either the inherent biases or difﬁculty to learn the parameters of the baselines, it is unclear how effective these newly proposed estimators are in backpropagating the gradients through a sequence of discrete variables (Yin et al., 2019b).
Revised: However, due to either the inherent biases or difﬁculty to learn the parameters of the baselines, it is unclear how effective these newly proposed estimators are in backpropagating the gradients through a sequence of discrete variables (Yin et al., 2019).

Original: This is exacerbated in contextual categorical sequence generation problems, where it is common for a sequence to contain quite a few tokens/actions, each of which is selected from a set of thousands of candidates.
Revised: This is exacerbated in contextual categorical sequence generation problems, where it is common for a sequence to contain quite a few tokens/actions, each of which is selected from a set of thousands of candidates.

Original: Another practical issue is that generating a token from a large vocabulary via the softmax output layer is often computationally heavy.
Revised: Another practical issue is that generating a token from a large vocabulary via the softmax output layer is often computationally heavy.

Original: This prevents a categorical sequence generation model from being deployed to low-power devices.
Revised: This prevents a categorical sequence generation model from being deployed to low-power devices.

Original: Despite signiﬁcant recent efforts in addressing the computation bottleneck due to a wide softmax layer (Shim et al., 2017; Zhang et al., 2018; Chen et al., 2018), for categorical sequence generation, it is so far unclear how to address the softmax computational bottleneck while at the same time providing low-variance gradient of its RL objective.
Revised: Despite signiﬁcant recent efforts in addressing the computation bottleneck due to a wide softmax layer (Shim et al., 2017; Zhang et al., 2018; Chen et al., 2018), for categorical sequence generation, it is so far unclear how to address the softmax computational bottleneck while at the same time providing low-variance gradient of its RL objective.

Original: This paper makes two primary contributions: 1) To address the high gradient variance issue, we adapt to contextual categorical sequence generation tasks the augment-REINFORCE-swap-merge (ARSM) estimator (Yin et al., 2019a), which provides unbiased, low-variance gradients for categorical variables, using token-level rewards from correlated MC rollouts that naturally serve as the baselines for each other.
Revised: This paper makes two primary contributions: 1) To address the high gradient variance issue, we adapt to contextual categorical sequence generation tasks the augment-REINFORCE-swap-merge (ARSM) estimator (Yin et al., 2019), which provides unbiased, low-variance gradients for categorical variables, using token-level rewards from correlated MC rollouts that naturally serve as the baselines for each other.

Original: We show that the number of rollouts is adapted to the model uncertainty on the latest generated token, and can also be manually controlled to balance variance reduction and computation.
Revised: We show that the number of rollouts is adapted to the model uncertainty on the latest generated token, and can also be manually controlled to balance variance reduction and computation.

Original: 2) To address the high generation cost issue, we replace the generation of each categorical variable in the sequence, via a categorical softmax output layer, with the generation of a sequence of binary decisions on a binary tree from its root node to a leaf node, which is occupied by a unique term of the vocabulary.
Revised: 2) To address the high generation cost issue, we replace the generation of each categorical variable in the sequence, via a categorical softmax output layer, with the generation of a sequence of binary decisions on a binary tree from its root node to a leaf node, which is occupied by a unique term of the vocabulary.

Original: For training, we adapt the augment-REINFORCE-merge (ARM) estimator (Yin & Zhou, 2019), which provides unbiased, low-variance gradients for binary variables, to backpropagate the gradients through the sequence of binary sequences.
Revised: For training, we adapt the augment-REINFORCE-merge (ARM) estimator (Yin & Zhou, 2019), which provides unbiased, low-variance gradients for binary variables, to backpropagate the gradients through the sequence of binary sequences.

Original: Under this binary-tree construction, the cost of generating a categorical token reduces from O ( V ) to O (log 2 ( V )) , where V is the vocabulary size.
Revised: Under this binary-tree construction, the cost of generating a categorical token reduces from O ( V ) to O (log 2 ( V )) , where V is the vocabulary size.

Original: We demonstrate our methods on two representative contextual categorical sequence generation tasks, with the number of actions ranging from 53 (neural program synthesis) to 9978 (image captioning).
Revised: We demonstrate our methods on two representative contextual categorical sequence generation tasks, with the number of actions ranging from 53 (neural program synthesis) to 9978 (image captioning).

Original:  P RELIMINARIES ON CONTEXTUAL SEQUENCE GENERATION
Revised:  P RELIMINARIES ON CONTEXTUAL SEQUENCE GENERATION

Original: For a dataset of context-output pairs D := { x i , y i } Ni =1 , our goal is to learn the conditional distribution of output y i given its context x i , expressed as p
Revised: For a dataset of context-output pairs D := { x i , y i } Ni =1 , our goal is to learn the conditional distribution of output y i given its context x i , expressed as p

Original: θ ( y i | x i ) .
Revised: θ ( y i | x i ) .

Original: Below we drop the data index subscript for brevity.
Revised: Below we drop the data index subscript for brevity.

Original: We focus on the case that an output is a sequence of T categorical variable as y =
Revised: We focus on the case that an output is a sequence of T categorical variable as y =

Original: { y 1 , · · · , y T } , where y t ∈ { 1 , . .
Revised: { y 1 , · · · , y T } , where y t ∈ { 1 , . .

Original: , V } .
Revised: , V } .

Original: A common way to model p θ ( y | x ) is to decompose it as p
Revised: A common way to model p θ ( y | x ) is to decompose it as p

Original: θ ( y | x ) =
Revised: θ ( y | x ) =

Original: (cid:81)
Revised: (cid:81)

Original: Tt =1 p θ ( y t | y 1: t − 1 , x ) , where the t -th term in the product, which models the distribution of token y t conditioning on the context x and previously generated tokens y 1: t − 1 , is commonly parameterized by a recurrent neural network (Sutskever et al., 2014).
Revised: Tt =1 p θ ( y t | y 1: t − 1 , x ) , where the t -th term in the product, which models the distribution of token y t conditioning on the context x and previously generated tokens y 1: t − 1 , is commonly parameterized by a recurrent neural network (Sutskever et al., 2014).

Original: MLE is a common way to train the model: ˆ θ MLE = argmax θ E { x , y }∼ p data ( x , y )
Revised: MLE is a common way to train the model: ˆ θ MLE = argmax θ E { x , y }∼ p data ( x , y )

Original: [log p θ ( y | x )] .
Revised: [log p θ ( y | x )] .

Original: Viewing p θ ( y t | y 1: t − 1 , x ) as a stochastic policy for choosing an action given the state, we can formulate contextual sequence generation as an RL problem and infer the policy parameter θ as ˆ θ RL = argmax θ E { x , y }∼ p data ( x
Revised: Viewing p θ ( y t | y 1: t − 1 , x ) as a stochastic policy for choosing an action given the state, we can formulate contextual sequence generation as an RL problem and infer the policy parameter θ as ˆ θ [Equation] = [Equation]

Original: , y ) E 
Revised: E [Equation]

Original: z ∼ p θ ( · | x ) [ r ( z | x , y )] , where r ( z | x , y ) denotes the reward of the generated (hypothesis) sequence z given the context x and the reference target sequence y .
Revised: p θ ( [Equation] | x ) [ r ( z | x , y )] , where r ( z | x , y ) denotes the reward of the generated (hypothesis) sequence z given the context x and the reference target sequence y .

Original: For example, for image captioning, the reward could be the CIDEr score that measures the similarity between the generated caption z and the reference y
Revised: For example, for image captioning, the reward could be the CIDEr score that measures the similarity between the generated caption z and the reference y

Original: (Rennie et al., 2017).
Revised: (Rennie et al., 2017).

Original: Denote σ ( · ) as the softmax function and T θ ( · ) as a deterministic function deﬁned by a deep neural network with parameter θ .
Revised: Denote σ ( · ) as the softmax function and T θ ( · ) as a deterministic function deﬁned by a deep neural network with parameter θ .

Original: We model p θ ( z | x ) = (cid:81)
Revised: We model p θ ( z | x ) = (cid:81)

Original: Tt =1 p θ ( z t | z 1: t − 1 , x ) , where
Revised: Tt =1 p θ ( z t | z 1: t − 1 , x ) , where

Original: [Equation]
Revised: [Equation]

Original: For a context-target pair { x , y } , we can expand the expected reward ER under policy p θ ( z | x ) as
Revised: For a context-target pair { x , y } , we can expand the expected reward ER under policy p θ ( z | x ) as

Original: [Equation]
Revised: [Equation]

Original: where the partial-sentence reward is deﬁned as r ( z1: t | x , y )
Revised: where the partial-sentence reward is deﬁned as r ( z

Original: =
Revised: [Equation]

Original: E z t+1: T ∼ p θ ( · | x,z
Revised: t | x , y ) = E [Equation]

Original: [Equation]
Revised: [Equation]

Original: t )
Revised: z

Original: [ r ( z | x , y )] .
Revised: [Equation] x , y )] .

Original: Using the chain rule and REINFORCE (Williams, 1992) estimator, we have
Revised: Using the chain rule and REINFORCE (Williams, 1992) estimator, we have

Original: [Equation]
Revised: [Equation]

Original: The main challenge here is to control the variance in estimating ∇ θ ER.
Revised: The main challenge here is to control the variance in estimating ∇ θ ER.

Original: A variety of methods have been proposed.
Revised: A variety of methods have been proposed.

Original: For example, drawing sentence z ∼ p θ ( z | x ) and using its reward r ( z | x , y ) to approximate all partial-sentence rewards { r ( z 1: t | x , y ) } 1 ,T , Ranzato et al.
Revised: For example, drawing sentence z ∼ p θ ( z | x ) and using its reward r ( z | x , y ) to approximate all partial-sentence rewards { r ( z 1: t | x , y ) } 1 ,T , Ranzato et al.

Original: (2016) introduce MIXER with a scheduled training iterating between MLE and RL, estimating the RL gradient as
Revised: (2016) introduce MIXER with a scheduled training iterating between MLE and RL, estimating the RL gradient as

Original: [Equation]
Revised: [Equation]

Original: where b ( z 1: t − 1 ) is a baseline function.
Revised: where b ( z 1: t − 1 ) is a baseline function.

Original: To improve MIXER, Rennie et al.
Revised: To improve MIXER, Rennie et al.

Original: (2017) introduces the self-critic (SC) sequence training algorithm that sets b ( z 1: t − 1 )
Revised: (2017) introduces the self-critic (SC) sequence training algorithm that sets b ( z 1: t − 1 )

Original: =
Revised: =

Original: r (˜ z | x , y ) for all t , where ˜ z is a greedy sequence rollout under p θ ( z | x ) .
Revised: r (˜ z | x , y ) for all t , where ˜ z is a greedy sequence rollout under p θ ( z | x ) .

Original: As using sentence-level reward r ( z | x , y ) to guide the learning is found to be sensitive to the algorithm parameters such as the learning rate, Liu et al.
Revised: As using sentence-level reward r ( z | x , y ) to guide the learning is found to be sensitive to the algorithm parameters such as the learning rate, Liu et al.

Original: (2017) follow Yu et al.
Revised: (2017) follow Yu et al.

Original: (2017) to use token-level rewards that approximate each partial-sentence reward with
Revised: (2017) to use token-level rewards that approximate each partial-sentence reward with

Original: K independent MC rollouts (MC- K ) as ˆ r
Revised: K independent MC rollouts (MC- K ) as ˆ r

Original: ( z 1: t | x , y )
Revised: ( z 1: t | x , y )

Original: = 1 K (cid:80)
Revised: = 1 K (cid:80)

Original: Kk =1 r ( z 1: t , z ( k ) ( t +1): T | x , y ) , z ( k ) ( t +1): T ∼ p θ ( · | x , z 1: t ) .
Revised: Kk =1 r ( z 1: t , z ( k ) ( t +1): T | x , y ) , z ( k ) ( t +1): T ∼ p θ ( · | x , z 1: t ) .

Original: With these token-level rewards ˆ r ( z 1: t | x , y ) , Liu et al.
Revised: With these token-level rewards ˆ r ( z 1: t | x , y ) , Liu et al.

Original: (2017) estimate the gradient as
Revised: (2017) estimate the gradient as

Original: [Equation]
Revised: [Equation]

Original: Following SC, for token t , one may choose baseline b ( z 1: t − 1 )
Revised: Following SC, for token t , one may choose baseline b ( z 1: t − 1 )

Original: =
Revised: =

Original: r ( z 1: t − 1 , ˜
Revised: r ( z 1: t − 1 , ˜

Original: z t : T | x , y ) , where ˜ z t : T is a greedy rollout following partial sentence z 1: t − 1 .
Revised: z t : T | x , y ) , where ˜ z t : T is a greedy rollout following partial sentence z 1: t − 1 .

Original: In addition to being more robust, using token-level rewards ˆ r ( z 1: t ) to guide the learning is often necessary when the reward signal is sparse, e.g. , in neural program synthesis, it is common that a full MC roll receives zero reward with high probability.
Revised: In addition to being more robust, using token-level rewards ˆ r ( z 1: t ) to guide the learning is often necessary when the reward signal is sparse, e.g. , in neural program synthesis, it is common that a full MC roll receives zero reward with high probability.

Original: In addition to these methods mentioned above, the actor-critic method (Sutton, 1988) is used to reduce gradient variance at the expense of introducing bias (Bahdanau et al., 2016; Zhang et al., 2017).
Revised: In addition to these methods mentioned above, the actor-critic method (Sutton, 1988) is used to reduce gradient variance at the expense of introducing bias (Bahdanau et al., 2016; Zhang et al., 2017).

Original: Several approaches explore beam search instead of sampling based methods (Wiseman & Rush, 2016;
Revised: Several approaches explore beam search instead of sampling based methods (Wiseman & Rush, 2016;

Original: Bunel et al., 2018), also at the expense of introducing bias.
Revised: Bunel et al., 2018), also at the expense of introducing bias.

Original: Several other methods combine the MLE and RL objectives for training (Norouzi et al., 2016; Ding & Soricut, 2017).
Revised: Several other methods combine the MLE and RL objectives for training (Norouzi et al., 2016; Ding & Soricut, 2017).

Original:  P OLICY GRADIENT WITH ADAPTIVE CORRELATED MC ROLLOUTS
Revised:  P OLICY GRADIENT WITH ADAPTIVE CORRELATED MC ROLLOUTS

Original: Correlated MC samples, if well designed, can be combined to achieve much greater variance reduction than using the same number of independent ones (Owen, 2013).
Revised: Correlated MC samples, if well designed, can be combined to achieve much greater variance reduction than using the same number of independent ones (Owen, 2013).

Original: To reduce gradient variance for contextual categorical sequence generation and remove the need to construct explicit baselines, we adapt the augment-REINFORCE-swap (ARS) and ARS-merge (ARSM) estimators of Yin et al.
Revised: To reduce gradient variance for contextual categorical sequence generation and remove the need to construct explicit baselines, we adapt the augment-REINFORCE-swap (ARS) and ARS-merge (ARSM) estimators of Yin et al.

Original: (2019b) to generate correlated MC rollouts.
Revised: (2019) to generate correlated MC rollouts.

Original: The number of correlated MC rollouts, used to estimate each token-level partial-sequence reward, is adaptive according to the uncertainty on token generation.
Revised: The number of correlated MC rollouts, used to estimate each token-level partial-sequence reward, is adaptive according to the uncertainty on token generation.

Original: The key idea here is to rewrite the gradient as differently expressed but equivalent expectations, whose MC samples, generated by sharing the same set of random numbers and hence correlated, are subsequently merged for variance reduction, without the need of learnable baseline functions.
Revised: The key idea here is to rewrite the gradient as differently expressed but equivalent expectations, whose MC samples, generated by sharing the same set of random numbers and hence correlated, are subsequently merged for variance reduction, without the need of learnable baseline functions.

Original: Denote z ∼ Cat ( σ ( φ )) as a univariate categorical variable such that P ( z = v | φ ) =
Revised: Denote z ∼ Cat ( σ ( φ )) as a univariate categorical variable such that P ( z = v | φ ) =

Original: σ ( φ ) v = e φ v (cid:14)(cid:80)[-V i =
-]
Revised: σ ( φ ) v = e φ v (cid:14)(cid:80)[-V i =
-]

Original:  e φ i .
Revised: Vi =1 e φ i .

Original: Denote π =
Revised: Denote π =

Original: ( π 1 , . . .
Revised: ( π 1 , . . .

Original: , π V ) ∼ Dir ( 1 V ) as a random probability vector drawn from the Dirichlet distribution whose V parameters are all ones.
Revised: , π V ) ∼ Dir ( 1 V ) as a random probability vector drawn from the Dirichlet distribution whose V parameters are all ones.

Original: With m (cid:11) jdenoted as the operation of swapping the m th and j th elements of a vector, we have π m (cid:11) j m = π j , π m (cid:11) j j = π m ,  π m (cid:11)j i =
Revised: Denoting m (cid:11) j as the operation of swapping the m th and j th elements of a vector, we have π m (cid:11) j m = π j , π m (cid:11) [Equation] = π m , and π m (cid:11)

Original:  π i , ∀ i / ∈ { m, j } .
Revised: j i = π i , ∀ i / ∈ { m, j } .

Original: Note with z := argmin i ∈{ 1 , ··· ,V } π i e − φ i , E ( φ ) =
Revised: With z := argmin i ∈{ 1 , ··· ,V } π i e − φ i , E ( φ ) =

Original: E c ∼ Cat ( σ ( φ ))
Revised: E c ∼ Cat ( σ ( φ ))

Original: [ r ( c )] can be reexpressed as E ( φ ) =
Revised: [ r ( c )] can be reexpressed as E ( φ ) =

Original: E π ∼ Dir ( 1 V )
Revised: E π ∼ Dir ( 1 V )

Original: [ r ( z )] , whose gradient under the ARS estimator can be expressed as∇
Revised: [ r ( z )] , whose gradient under the ARS estimator can be expressed as

Original: [Equation]
Revised: [Equation]

Original: where j is a reference category randomly selected from { 1 , . .
Revised: where j is a reference category randomly selected from { 1 , . .

Original: , V } and z m (cid:11)
Revised: , V } and z m (cid:11)

Original: j := argmin i ∈{ 1 , ··· ,V } π m (cid:11)
Revised: j := argmin i ∈{ 1 , ··· ,V } π m (cid:11)

Original: j i e − φ i .
Revised: j i e [Equation] i .

Original: ARSM further improves ARS by adding a merge step as
Revised: ARSM further improves ARS by adding a merge step as

Original: [Equation]
Revised: [Equation]

Original: We refer to z as the true action and z m (cid:11) j as pseudo actions.
Revised: We refer to z as the true action and z m (cid:11) j as pseudo actions.

Original: These actions are correlated to each other as they are transformed from the same Dirichlet distributed π vector under different pairwise index swaps.
Revised: These actions are correlated to each other as they are transformed from the same Dirichlet distributed π vector under different pairwise index swaps.

Original: While there are V ( V − 1) / 2 unique pairwise swaps, after MLE pre-train with V ∼ 10 4 , the number of unique pseudo actions that differ from the true action is random and often stays below 10, and becomes 0 more and more frequently as the progress of RL training reduces model uncertainty.
Revised: While there are V ( V − 1) / 2 unique pairwise swaps, after MLE pre-train with V ∼ 10 4 , the number of unique pseudo actions that differ from the true action is random and often stays below 10, and becomes 0 more and more frequently as the progress of RL training reduces model uncertainty.

Original: 3. A DAPTIVECORRELATED MC BASED POLICY GRADIENT FOR CATEGORICAL SEQUENCE
Revised: 3. A DAPTIVE

Original: Applying the ARSM estimator in (5) to the expected reward shown in (2), we have 
Revised: Applying the ARSM estimator in (5) to the expected reward shown in (2), we have ∇

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  where z m (cid:11) j t :=
Revised: , where z m (cid:11) j t :

Original: [Equation]
Revised: [Equation]

Original: i ∈{

Original: [Equation]
Revised: [Equation]

Revised: , π

Original: [Equation]
Revised: [Equation]

Revised: ∼ Dir ( 1 V ) , and r ( z 1:

Original: [Equation]
Revised: [Equation]

Original: } π
Revised: 1 , z

Original: [Equation]
Revised: [Equation]

Original: , π t ∼ Dir ( 1 V ) , and r ( z 1: t − 1 , z m (cid:11) j t | x , y ) = 
Revised: j t | x , y ) = E z [Equation] | x ,z

Original: E z ( t +1): T ∼ p θ ( · | x ,z 1: t − 1 ,z m (cid:11) j t )
Revised: [Equation] [Equation] z [Equation]

Original: [ r ( z | x , y )] .
Revised: [ r ( z | x , y )] .

Original: We can therefore estimate each expectation using regular MC in the augmented space.
Revised: We can therefore estimate each expectation using regular MC in the augmented space.

Original: Detailed formulations are deferred to Appendix B.1.
Revised: Detailed formulations are deferred to Appendix B.1.

Original: Note if given π t , all pseudo actions z m (cid:11) j t are equal to true action z t , then g ARSM ( π t ) v = g ARS ( π t , j ) v = 0 .
Revised: Note if given π t , all pseudo actions z m (cid:11) j t are equal to true action z t , then g ARSM ( π t ) v = g ARS ( π t , j ) v = 0 .

Original: We note while the notation appears cumbersome, its implementation is not difﬁcult, as described in
Revised: We note while the notation appears cumbersome, its implementation is not difﬁcult, as described in

Original: Algorithm 2, Appendix C.
Revised: Algorithm 2, Appendix C.

Original: The intuitive explanation of ARSM is that given π 1: T , it ﬁrst generates true action sequence z 1: T (main trajectory) with z t = argmin
Revised: The intuitive explanation of ARSM is that given π 1: T , it ﬁrst generates true action sequence z 1: T (main trajectory) with z t = argmin

Original: i π ti e φ ti ; it then performs embarrassingly parallel MC rollouts for all unique pseudo actions that differ from their corresponding true actions: at step t , given the true actions z 1: t − 1 , it generates pseudo actions z m (cid:11) j t , and for each unique value of them that differs from z t , it estimates its expected reward by rolling out a full sequence of length T ; and ﬁnally it combines the sampled rewards of the true action sequence and unique pseudo action sequences, which are correlated to each other, to achieve signiﬁcant variance reduction.
Revised: i π ti e φ ti ; it then performs embarrassingly parallel MC rollouts for all unique pseudo actions that differ from their corresponding true actions: at step t , given the true actions z 1: t − 1 , it generates pseudo actions z m (cid:11) j t , and for each unique value of them that differs from z t , it estimates its expected reward by rolling out a full sequence of length T ; and ﬁnally it combines the sampled rewards of the true action sequence and unique pseudo action sequences, which are correlated to each other, to achieve signiﬁcant variance reduction.

Original: Despite signiﬁcant gradient variance reduction, ARSM may become less efﬁcient in computation when V becomes large ( e.g., ∼ 10 , 000 ).
Revised: Despite signiﬁcant gradient variance reduction, ARSM may become less efﬁcient in computation when V becomes large ( e.g., ∼ 10 , 000 ).

Original: In the worst case, for each gradient estimate, it needs to generate as many as V − 1 unique pseudo action sequences at each token; while in practice, the actual number is much smaller, it still could be large enough to cause computational issues, especially if the policy parameter is far from convergence.
Revised: In the worst case, for each gradient estimate, it needs to generate as many as V − 1 unique pseudo action sequences at each token; while in practice, the actual number is much smaller, it still could be large enough to cause computational issues, especially if the policy parameter is far from convergence.

Original: We note while in theory ARSM enjoys embarrassingly parallel computation for rolling out all unique pseudo action sequences, the acceleration via parallelization in practice is constrained by the capacity of our own computation platform.
Revised: We note while in theory ARSM enjoys embarrassingly parallel computation for rolling out all unique pseudo action sequences, the acceleration via parallelization in practice is constrained by the capacity of our own computation platform.

Original: This motivates the following remedy.
Revised: This motivates the following remedy.

Original: For large V , we choose K reference categories γ 1 , . . .
Revised: For large V , we choose K reference categories γ 1 , . . .

Original: , γ K , randomly sampled from { 1 , . . .
Revised: , γ K , randomly sampled from { 1 , . . .

Original: , V } without replacement, to perform the swapping operations for pseudo action generation, and averaging over their corresponding ARS estimators as
Revised: , V } without replacement, to perform the swapping operations for pseudo action generation, and averaging over their corresponding ARS estimators as

Original: [Equation]
Revised: [Equation]

Original: We refer to this gradient estimator as the ARS-K gradient estimator.
Revised: We refer to this gradient estimator as the ARS-K gradient estimator.

Original: Whether this remedy could be successful depends on how large K needs to be as V increases.
Revised: Whether this remedy could be successful depends on how large K needs to be as V increases.

Original: We ﬁnd via experiments that the sufﬁcient size of K grows slowly as V increases.
Revised: We ﬁnd via experiments that the sufﬁcient size of K grows slowly as V increases.

Original: For example, we will show in Section 4.2 that for the image captioning task with V = 9 , 788 , setting K = 5 already leads to competitive results.
Revised: For example, we will show in Section 4.2 that for the image captioning task with V = 9 , 788 , setting K = 5 already leads to competitive results.

Original: Note during testing, regardless of whether using ARSM, ARS-K, or some other estimators, the categorical softmax output layer could become the computation bottleneck for random sequence generation.
Revised: Note during testing, regardless of whether using ARSM, ARS-K, or some other estimators, the categorical softmax output layer could become the computation bottleneck for random sequence generation.

Original: This motivates us to provide an algorithm to signiﬁcantly reduce the generation cost during testing, though at the expense of reduced performance.
Revised: This motivates us to provide an algorithm to considerably reduce the generation cost during testing, though at the expense of reduced performance.

Original: We describe such a solution below.
Revised: We describe such a solution below.

Original: 3. B INARY - TREE -ARSM FOR COMPUTATIONAL RESOURCE LIMITED APPLICATIONS
Revised: 3. B INARY - TREE -ARSM FOR COMPUTATIONAL RESOURCE LIMITED APPLICATIONS

Original: The conventional way to generate a word token is to sample from a V -way categorical distribution, whose probability parameters are obtained via a softmax output layer.
Revised: The conventional way to generate a word token is to sample from a V -way categorical distribution, whose probability parameters are obtained via a softmax output layer.

Original: This softmax output layer often becomes the computation bottleneck when V is large, making it difﬁcult to be applied to resource-constrained environments, such as mobile devices.
Revised: This softmax output layer often becomes the computation bottleneck when V is large, making it difﬁcult to be applied to resource-constrained environments, such as mobile devices.

Original: To mitigate this issue, related to the hierarchical softmax idea (Morin & Bengio, 2005; Grave et al., 2017; Goodman, 2001), we ﬁrst construct a binary tree to allocate each word of the vocabulary to one and only one leaf node of this tree.
Revised: To mitigate this issue, related to the hierarchical softmax idea (Morin & Bengio, 2005; Grave et al., 2017; Goodman, 2001), we ﬁrst construct a binary tree to allocate each word of the vocabulary to one and only one leaf node of this tree.

Original: A simple solution is to perform binary hierarchical clustering of the words.
Revised: A simple solution is to perform binary hierarchical clustering of the words.

Original: Denote e v as the word embedding vector of word v .
Revised: Denote e v as the word embedding vector of word v .

Original: In this paper, we use agglomerative clustering (Sibson, 1973) on e 1 , . . .
Revised: In this paper, we use agglomerative clustering (Sibson, 1973) on e 1 , . . .

Original: , e V to recursively merge two closest clusters at a time until there is only one cluster.
Revised: , e V to recursively merge two closest clusters at a time until there is only one cluster.

Original: The root is linked to V leaf nodes via V overlapping root-to-leaf paths, each of which can be represented by a unique binary code b v of length D , where D =
Revised: The root is linked to V leaf nodes via V overlapping root-to-leaf paths, each of which can be represented by a unique binary code b v of length D , where D =

Original: O (log 2 V ) is the depth of the tree.
Revised: O (log 2 V ) is the depth of the tree.

Original: Note the V paths are not restricted to travel through the same number of nodes, but for simplicity we zero pad them to the same length.
Revised: Note the V paths are not restricted to travel through the same number of nodes, but for simplicity we zero pad them to the same length.

Original: Both off-the-shelf  (Pennington et al., 2014) and task-speciﬁc embedding vectors can be utilized.
Revised: Both off-the-shelf embedding vectors (Pennington et al., 2014) and task-speciﬁc ones can be utilized.

Original: They provide useful prior information about the structure of the vocabulary, which we can exploit to facilitate our search within the action space.
Revised: They provide useful prior information about the structure of the vocabulary, which we can exploit to facilitate our search within the action space.

Original: With the binary tree, we transform the problem of choosing one out of V categories into that of making a sequence of binary decisions b
Revised: With the binary tree, we transform the problem of choosing one out of V categories into that of making a sequence of binary decisions b

Original: = ( b 1 , . . .
Revised: = ( b 1 , . . .

Original: , b D ) .
Revised: , b D ) .

Original: If making l < D binary decisions ( b 1 , . . .
Revised: If making l < D binary decisions ( b 1 , . . .

Original: , b l ) has already led to a leaf node, then the sequence is terminated and b l +1 , . . .
Revised: , b l ) has already led to a leaf node, then the sequence is terminated and b l +1 , . . .

Original: , b D all become zeros.
Revised: , b D all become zeros.

Original: There is a one-to-one mapping between the V root-to-leaf paths and V vocabulary words.
Revised: There is a one-to-one mapping between the V root-to-leaf paths and V vocabulary words.

Original: We denote ν ( b ) ∈ { 1 , . .
Revised: We denote ν ( b ) ∈ { 1 , . .

Original: , V } as the word that path b is mapped to, and β ( v ) ∈ { 0 , 1 } D as the path that word v is mapped to.
Revised: , V } as the word that path b is mapped to, and β ( v ) ∈ { 0 , 1 } D as the path that word v is mapped to.

Original: Note for a binary tree with V leaves, there will be V − 1 non-leaf nodes, each of which needs a logit φ for its Bernoulli probability.
Revised: Note for a binary tree with V leaves, there will be V − 1 non-leaf nodes, each of which needs a logit φ for its Bernoulli probability.

Original: Thus in total we need V − 1 logits φ 1 , · · · , φ V − 1 .
Revised: Thus in total we need V − 1 logits φ 1 , · · · , φ V − 1 .

Original: The computational saving in generating categorical sequences comes from the fact that to generate a word token
Revised: The computational saving in generating categorical sequences comes from the fact that to generate a word token

Original: we need D φ ’s at most rather than all V − 1 φ ’s.
Revised: we need D φ ’s at most rather than all V − 1 φ ’s.

Original: Therefore, with the binary tree, the computation for the softmax output layer to generate a token decreases from O ( V ) to O (log 2 V ) , which is signiﬁcant especially for mobile applications.
Revised: Therefore, with the binary tree, the computation for the softmax output layer to generate a token decreases from O ( V ) to O (log 2 V ) , which is signiﬁcant especially for mobile applications.

Original: The binary-tree softmax model can be trained with MLE, or with a novel binary-tree-ARSM (BT-ARSM) gradient estimator introduced below.
Revised: The binary-tree softmax model can be trained with MLE, or with the binary-tree-ARSM (BT-ARSM) gradient estimator introduced below.

Original: For the binary case, both ARS and ARSM reduce to augment-REINFORCE-merge (ARM) (Yin &
Revised: For the binary case, both ARS and ARSM reduce to augment-REINFORCE-merge (ARM) (Yin &

Original: Zhou, 2019), which expresses the gradient of E b ( φ ) =
Revised: Zhou, 2019), which expresses the gradient of E b ( φ ) =

Original: E z ∼ Ber ( σ ( φ ))
Revised: E z ∼ Ber ( σ ( φ ))

Original: [ r ( z )] , σ ( φ ) =
Revised: [ r ( z )] , σ ( φ ) =

Original: 1 / (1 + e − [Equation] ) , as
Revised: 1 / (1 + e − φ ) , as

Original: [Equation]
Revised: [Equation]

Original: where b true := 1 [ π<σ ( φ )] and b sudo := 1 [ π>σ ( − φ )] are referred to as the true and pseudo actions, respectively.
Revised: where b true := 1 [ π<σ ( φ )] and b sudo := 1 [ π>σ ( − φ )] are referred to as the true and pseudo actions, respectively.

Original: We note if we represent a V -way categorical variable as a sequence of D
Revised: We note if we represent a V -way categorical variable as a sequence of D

Original: = O
Revised: = O

Original: (log 2 V ) binary variables, the number of unique pseudo actions that differ from the true actions is at most D .
Revised: (log 2 V ) binary variables, the number of unique pseudo actions that differ from the true actions is at most D .

Original: In the binary-tree setting, the conditional probability of generating token z t is changed from (1) to p θ ( zt | x , z [Equation] ) =
Revised: In the binary-tree setting, the conditional probability of generating token z t is changed from (1) to p θ ( z

Original: [Equation]
Revised: [Equation]

Original: Bernoulli [Equation] [Equation] :=
Revised: | x , z

Original: [Equation]
Revised: [Equation]

Original: (8) where ( b t 1 , . . .
Revised: where ( b t 1 , . . .

Original: , b tD zt ) :
Revised: , b tD zt ) :

Original: = β ( z t ) , φ t,b t (1: l − 1) is the parameter of the non-leaf node at the end of the path deﬁned by b t (1: l − 1) , and D z t is the number of non-leaf nodes in the root-to-leaf path that leads to z t .
Revised: = β ( z t ) , φ t,b t (1: l − 1) is the parameter of the non-leaf node at the end of the path deﬁned by b t (1: l − 1) , and D z t is the number of non-leaf nodes in the root-to-leaf path that leads to z t .

Original: Similar to the derivation in Section 3.1, we apply the ARSM gradient estimation to the decomposed binary sequences (BT-ARSM).
Revised: Similar to the derivation in Section 3.1, we apply the ARSM gradient estimation to the decomposed binary sequences (BT-ARSM).

Original: We provide the detailed formulations in Appendix B.2 and pseudo code in Algorithm 3, Appendix C.
Revised: We provide the detailed formulations in Appendix B.2 and pseudo code in Algorithm 3, Appendix C.

Original: Intuitively, it ﬁrst samples the true sequence of binary sequences { ( b t 1 , . . .
Revised: Intuitively, it ﬁrst samples the true sequence of binary sequences { ( b t 1 , . . .

Original: , b tD zt ) }
Revised: , b tD zt ) }

Original: Tt =1 ; it then performs embarrassingly parallel MC rollouts for all pseudo actions that differ from their corresponding true actions: at step t , and depth l , given the previous true tokens z 1: t − 1 , and true binary code b t 1 , . . .
Revised: Tt =1 ; it then performs embarrassingly parallel MC rollouts for all pseudo actions that differ from their corresponding true actions: at step t , and depth l , given the previous true tokens z 1: t − 1 , and true binary code b t 1 , . . .

Original: , b t,l − 1 , it generates pseudo binary code b (sudo) tl , and if it differs from b tl , then we estimate its expected reward by ﬁrst rolling out a full binary code up to depth D and rolling out a full sequence up to length T ; and ﬁnally it combines the sampled rewards of the true action sequence and pseudo action sequences, which are correlated to each other, to achieve signiﬁcant variance reduction.
Revised: , b t,l − 1 , it generates pseudo binary code b (sudo) tl , and if it differs from b tl , then we estimate its expected reward by ﬁrst rolling out a full binary code up to depth D and rolling out a full sequence up to length T ; and ﬁnally it combines the sampled rewards of the true action sequence and pseudo action sequences, which are correlated to each other, to achieve signiﬁcant variance reduction.

Original: Note that if b (sudo) tl is the same as b tl , then the corresponding ARM gradient is zero.
Revised: Note that if b (sudo) tl is the same as b tl , then the corresponding ARM gradient is zero.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our models with both neural program synthesis (NPS) and image captioning.
Revised: We evaluate our models with both neural program synthesis (NPS) and image captioning.

Original: The source code implemented in PyTorch can be found at the following anonymous link: https://drive.google.com/ﬁle/d/1Af53I2DG-8dQGffLKuFbO2yUGJOGTxsv/view?usp=sharing.

Original: [Table]
Revised: [Table]

Original: 4. N EURAL PROGRAM SYNTHESIS
Revised: 4. N EURAL PROGRAM SYNTHESIS

Original: NPS is a challenging representative task in contextual categorical sequence generation.
Revised: NPS is a challenging representative task in contextual categorical sequence generation.

Original: First, the reward is only available after ﬁnishing the whole sequence.
Revised: First, the reward is only available after ﬁnishing the whole sequence.

Original: Second, the initial reward signals are often sparse because the generated programs rarely succeed in the beginning of training.
Revised: Second, the initial reward signals are often sparse because the generated programs rarely succeed in the beginning of training.

Original: We follow
Revised: We follow

Original: Bunel et al.
Revised: Bunel et al.

Original: (2018) to investigate an NPS task: for data sample i consisting of a set of input-output states { I mi , O mi } m =1 ,M i , the goal is to learn a synthesizer parameterized by θ to generate a program
Revised: (2018) to investigate an NPS task: for data sample i consisting of a set of input-output states { I mi , O mi } m =1 ,M i , the goal is to learn a synthesizer parameterized by θ to generate a program

Original: λ i , which will produce a sequence of categorical actions to map input state I mi to output state O mi ( i.e. , λ i ( I mi ) =
Revised: λ i , which will produce a sequence of categorical actions to map input state I mi to output state O mi ( i.e. , λ i ( I mi ) =

Original: O mi ) for all m ∈ { 1 , . . .
Revised: O mi ) for all m ∈ { 1 , . . .

Original: , M i } .
Revised: , M i } .

Original: The evaluation metric is Generalization (Bunel et al., 2018), deﬁned as the proportion of the test instances { I mi (cid:48) , O mi (cid:48) } m =1 ,M i that satisfy λ i (cid:48)
Revised: The evaluation metric is Generalization (Bunel et al., 2018), deﬁned as the proportion of the test instances { I mi (cid:48) , O mi (cid:48) } m =1 ,M i that satisfy λ i [Table]

Original: ( I mi (cid:48) )
Revised: ( I mi (cid:48) )

Original: =
Revised: =

Original: O mi (cid:48) for all m ∈ { 1 , ..., M i } .
Revised: O mi [Table] for all m ∈ { 1 , ..., M i } .

Original: We evaluate on the Karel dataset (Devlin et al., 2017a), consisting of 10 ,training reference Karel programs 1 with 2 , 500 validation and 2 , 500 test samples.
Revised: We evaluate on the Karel dataset (Devlin et al., 2017a), consisting of 10 ,training reference Karel programs 1 with 2 , 500 validation and 2 , 500 test samples.

Original: Each program consists of a sequence of actions to move an agent inside a grid-world from one starting grid (input) to an end grid (output).
Revised: Each program consists of a sequence of actions to move an agent inside a grid-world from one starting grid (input) to an end grid (output).

Original: The size of the action space V is 53 and average program length is around 20 .
Revised: The size of the action space V is 53 and average program length is around 20 .

Original: Baselines We incorporate ﬁve baseline algorithms in our evaluation.
Revised: Baselines We incorporate ﬁve baseline algorithms in our evaluation.

Original: ( i ) MC-2 (Eq 3), using tokenlevel rewards and greedy baselines.
Revised: ( i ) MC-2 (Eq 3), using tokenlevel rewards and greedy baselines.

Original: ( ii ) MC-0 , using sentence-level reward and token-level greedy baselines, which corresponds to the TD-SCST in Rennie et al.
Revised: ( ii ) MC-0 , using sentence-level reward and token-level greedy baselines, which corresponds to the TD-SCST in Rennie et al.

Original: ( iii ) REINFORCE , using sentence-level reward and with mini-batch mean as the baseline.
Revised: ( iii ) REINFORCE , using sentence-level reward and with mini-batch mean as the baseline.

Original: ( iv ) Self-Critic (SC) as in Rennie et al.
Revised: ( iv ) Self-Critic (SC) as in Rennie et al.

Original: ( v ) RL beam , the state-of-the-art method for NPS proposed by Bunel et al.
Revised: ( v ) RL beam , the state-of-the-art method for NPS proposed by Bunel et al.

Original: (2018) to reduce the gradient variance while sacriﬁcing the unbiasedness.
Revised: (2018) to reduce the gradient variance while sacriﬁcing the unbiasedness.

Original: The objective of RL beam is to maximize the expected reward under a distribution deﬁned on a space constructed with beam search
Revised: The objective of RL beam is to maximize the expected reward under a distribution deﬁned on a space constructed with beam search

Original: BS ( p θ , S ) , where S = 64 is the beam size.
Revised: BS ( p θ , S ) , where S = 64 is the beam size.

Original: Since the vocabulary size of V = 53 is not that large, we directly apply ARSM ( i.e. , ARS-53) policy gradient and compare it with the other methods.
Revised: Since the vocabulary size of V = 53 is not that large, we directly apply ARSM ( i.e. , ARS-53) policy gradient and compare it with the other methods.

Original: We use the code of Bunel et al.
Revised: We use the code of Bunel et al.

Original: as basis and use the same model architecture except for the exclusion of the optional grammar checker.
Revised: as basis and use the same model architecture except for the exclusion of the optional grammar checker.

Original: The grammar checker, not available for all NPS tasks, helps adaptively reduce the search (action) space and hence simpliﬁes optimization.
Revised: The grammar checker, not available for all NPS tasks, helps adaptively reduce the search (action) space and hence simpliﬁes optimization.

Original: Excluding the optional grammar checker eliminates its confounding inﬂuence on the core NPS task, making the comparison more generic and fair.
Revised: Excluding the optional grammar checker eliminates its confounding inﬂuence on the core NPS task, making the comparison more generic and fair.

Original: We use greedy search for both testing and validation.
Revised: We use greedy search for both testing and validation.

Original: All policy gradient based methods are ﬁne-tuning a pre-trained (and converged) MLE model.
Revised: All policy gradient based methods are ﬁne-tuning a pre-trained (and converged) MLE model.

Original: Results and analysis Figs.
Revised: Results and analysis Figs.

Original: 1a and 1b plot against iteration the log variances, and average number of rollouts (including greedy rollouts used to construct baselines) per step for each method.
Revised: 1a and 1b plot against iteration the log variance, and average number of rollouts (including greedy rollouts used to construct baselines) per step for each method.

Original: We observe that ARSM overall has the smallest gradient variance, and at the beginning ARSM has more
Revised: We observe that ARSM overall has the smallest gradient variance, and at the beginning ARSM has more

Original: MC rollouts (unique pseudo actions) and hence takes relative longer time per iteration, but soon it becomes more and more conﬁdent (reﬂected as fewer and fewer pseudo actions per iteration) and turns faster.
Revised: MC rollouts (unique pseudo actions) and hence takes relative longer time per iteration, but soon it becomes more and more conﬁdent (reﬂected as fewer and fewer pseudo actions per iteration) and turns faster.

Original: We note that the gradient variance at a given iteration is related to both the property of the gradient estimator and the parameter value at that iteration.
Revised: We note that the gradient variance at a given iteration is related to both the property of the gradient estimator and the parameter value at that iteration.

Original: Thus having smaller gradient variance may not necessarily imply better performance if different learning algorithms are not moving their parameters towards the same solution.
Revised: Thus having smaller gradient variance may not necessarily imply better performance if different learning algorithms are not moving their parameters towards the same solution.

Original: This could help explain why Self-Critic has lower gradient variance than both MC-0 and MC-2 do but worse validation and test Generalization scores.
Revised: This could help explain why SC has lower gradient variance than both MC-0 and MC-2 do but worse validation and test Generalization scores.

Original: Figs. 1c and 1d plot the Generalization scores against training time on the training and validation sets.
Revised: Figs. 1c and 1d plot the Generalization scores against training time on the training and validation sets.

Original: Due to large gradient variance, all methods except ARSM and RL beam either diverge or fail to improve the training objective.
Revised: Due to large gradient variance, all methods except ARSM and RL beam either diverge or fail to improve the training objective.

Original: Examining the performance on the training and validation sets suggests that REINFORCE and Self-Critic both diverge quickly; MC-0 stays around the starting point; MC-2 improves upon MLE initially, but then gradually diverges; RL beam reaches a good solution very fast but then gradually degrades towards worse solutions; and ARSM is the only one that makes steady improvement as the training progresses.
Revised: Examining the performance on the training and validation sets suggests that REINFORCE and SC both diverge quickly; MC-0 stays around the starting point; MCimproves upon MLE initially, but then gradually diverges; RL beam reaches a good solution very fast but then gradually degrades towards worse solutions; and ARSM is the only one that makes steady improvement as the training progresses.

Original: Observing how the gap between training and testing evolves, we see evidence suggesting that RL beam overﬁts the training data, possibly due to the use of biased gradients, while ARSM does not.
Revised: Observing how the gap between training and testing evolves, we see evidence suggesting that RL beam overﬁts the training data, possibly due to the use of biased gradients, while ARSM does not.

Original: We note that there is no explicit 
Revised: We note that there is no explicit regularization in

Original: [Figure]
Revised: [Figure]

Original: regularization in ARSM. However, since ARSM tends to generate fewer  unique pseudo actions as the policy becomes more  conﬁdent, this adaptive characteristic may serve as an implicit regularization during the training process.
Revised: ARSM. However, since ARSM tends to generate fewer and fewer unique pseudo actions as the policy becomes more and more conﬁdent, this adaptive characteristic may serve as an implicit regularization during the training process.

Original: Moreover, as the policy becomes more conﬁdent, the ARSM estimator has an increasing probability to yield MC gradient estimates that are exactly zeros, which may also help prevent overﬁtting as zero gradients will freeze the update of model parameters.
Revised: Moreover, as the policy becomes more conﬁdent, the ARSM estimator has an increasing probability to yield MC gradient estimates that are exactly zeros, which may also help prevent overﬁtting as zero gradients will freeze the update of model parameters.

Original: We summarize the validation and test Generation scores in Table 1.
Revised: We summarize the validation and test Generation scores in Table 1.

Original: Both MLE and RL beam (Bunel et al., 2018) perform reasonably well, but are outperformed by ARSM with a large margin.
Revised: Both MLE and RL beam (Bunel et al., 2018) perform reasonably well, but are outperformed by ARSM with a large margin.

Original: Even though MC-2 seems to improve upon MC-0 and SC, indicating the importance of using token-level rewards rather than sentence-level reward to guide the learning in this sparse reward scenario, it still clearly underperforms ARSM, which on average uses much fewer rollouts to estimate token-level rewards.
Revised: Even though MC-2 seems to improve upon MC-0 and SC, indicating the importance of using token-level rewards rather than sentence-level reward to guide the learning in this sparse reward scenario, it still clearly underperforms ARSM, which on average uses much fewer rollouts to estimate token-level rewards.

Original: This demonstrates the advantage of using an adaptive number of correlated MC rollouts over a ﬁxed number of independent MC rollouts.
Revised: This demonstrates the advantage of using an adaptive number of correlated MC rollouts over a ﬁxed number of independent MC rollouts.

Original: 4. I MAGE CAPTIONING
Revised: 4. I MAGE CAPTIONING

Original: Image captioning, mapping an image x to a summary sentence y =
Revised: Image captioning, mapping an image x to a summary sentence y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y T ) , has become a standard task to compare different policy gradient based RL methods.
Revised: , y T ) , has become a standard task to compare different policy gradient based RL methods.

Original: We conduct our experiments on the MS COCO dataset (Lin et al., 2014), following the standard data split from Karpathy & Fei-Fei (2015).
Revised: We conduct our experiments on the MS COCO dataset (Lin et al., 2014), following the standard data split from Karpathy & Fei-Fei (2015).

Original: We ﬁne-tune a pre-trained MLE model using CIDEr score as the reward.
Revised: We ﬁne-tune a pre-trained MLE model using CIDEr score as the reward.

Original: Our implementation is based on Luo et al.
Revised: Our implementation is based on Luo et al.

Original: Details about the experimental setup can be found in Appendix D.
Revised: Details about the experimental setup can be found in Appendix D.

Original: ARS-K for computation-sufﬁcient deployments We ﬁrst investigate the effectiveness of the proposed method when it is computationally feasible to use the categorical softmax output layer at the test time.
Revised: ARS-K for computation-sufﬁcient deployments We ﬁrst investigate the effectiveness of the proposed method when it is computationally feasible to use the categorical softmax output layer at the test time.

Original: We consider MLE, REINFORCE, Self-Critic (SC), and ARS-K with the same vocabulary size of V = 9788 .
Revised: We consider MLE, REINFORCE, SC, and ARS-K with the same vocabulary size of V = 9788 .

Original: For ARS-K, we experiment with several different K values.
Revised: For ARS-K, we experiment with several different K values.

Original: We report CIDEr score, and other commonly used metrics for the test set in Table 2.
Revised: We report CIDEr score, and other commonly used metrics for the test set in Table 2.

Original: We observe that while ARS-1 underperforms SC, ARS-K quickly improves as K increases: ARS-5 becomes comparable to SC in performance; ARS-10 and ARS-20 outperform SC by a large margin with statistical signiﬁcance (standard error is about 0 . 2 ).
Revised: We observe that while ARS-1 underperforms SC, ARS-K quickly improves as K increases: ARS-5 becomes comparable to SC in performance; ARS-10 and ARS-20 outperform SC by a large margin with statistical signiﬁcance (standard error is about 0 . 2 ).

Original: The superior performance of ARS-K with large K is also evidenced by Fig.
Revised: The superior performance of ARS-K with large K is also evidenced by Fig.

Original: The gradient variance of ARS-20 is signiﬁcantly lower than other algorithms (Fig.
Revised: The gradient variance of ARS-20 is signiﬁcantly lower than other algorithms (Fig.

Original: 3b upper).
Revised: 3b upper).

Original: In Fig.
Revised: In Fig.

Original: 3c (upper), we compare the average number of correlated MC rollouts of ARS-K for different K .
Revised: 3c (upper), we compare the average number of correlated MC rollouts of ARS-K for different K .

Original: While in theory the number of unique pseudo actions in ARS-K could be as many as V − 1 at each step, it can be seen that after MLE pre-training, for each ARS-K ( K = 1 , 5 , 10 , 20 ), on average that number is small (fewer than 10 for V = 9488 and K = 20 ) and has an evident decreasing trend during training.
Revised: While in theory the number of unique pseudo actions in ARS-K could be as many as V − 1 at each step, it can be seen that after MLE pre-training, for each ARS-K ( K = 1 , 5 , 10 , 20 ), on average that number is small (fewer than 10 for V = 9488 and K = 20 ) and has an evident decreasing trend during training.

Original: Moreover, it increases slowly as K increases (clearly below a linear increasing rate).
Revised: Moreover, it increases slowly as K increases (clearly below a linear increasing rate).

Original: Fig.
Revised: Fig.

Original: 2 shows two pictures  with their main sentences, which are greedily generated, and pseudo sentences generated by ARS-5 starting from the 7th token and 3rd token, respectively.
Revised: 2 shows two pictures (see more plots in Appendix E.1) with their main sentences, which are greedily generated, and pseudo sentences generated by ARS-5 starting from the 7th token and 3rd token, respectively.

Original: Both greedily 
Revised: Both greedily generated captions contain incorrect information about given

Original: [Figure]
Revised: [Figure]

Original: generated captions contain incorrect information about given images, while the pseudo sentences are semantically close to the greedy generations, however with interpretable variations in some details.
Revised: images, while the pseudo sentences are semantically close to the greedy generations, however with interpretable variations in some details.

Original: Some pseudo sentences are better than the greedily generated captions.
Revised: Some pseudo sentences are better than the greedily generated captions.

Original: These pseudo-captions assembled together capture the nuance variations of the neighborhood of the generation, thus can serve as a good baseline to reduce the variance of the policy gradient.
Revised: These pseudo-captions assembled together capture the nuance variations of the neighborhood of the generation, thus can serve as a good baseline to reduce the variance of the policy gradient.

Original: Note that there are more pseudo actions in the second plot, because the image is more complicated and also there is more uncertainty at the beginning stage ofa generation (3rd token) compared to the latter stage of generation (7th token).
Revised: Note that there are more pseudo actions in the second plot, because the image is more complex and also there is more uncertainty at the beginning stage of generation (3rd token) compared to the latter stage of generation (7th token).

Original: BT-ARSM for computation-limited deployments We evaluate the binary-tree ARSM (BT-ARSM) described in Section 3.2, which decomposes the action space to a sequence of binary actions.
Revised: BT-ARSM for computation-limited deployments We evaluate the binary-tree ARSM (BT-ARSM) described in Section 3.2, which decomposes the action space to a sequence of binary actions.

Original: (a) Binary tree constructions and MLE pretraining .
Revised: (a) Binary tree constructions and MLE pretraining .

Original: We explore three different ways to construct binary trees over the action space: ( i ) tree WV :
Revised: We explore three different ways to construct binary trees over the action space: ( i ) tree WV :

Original: We apply agglomerative clustering to off-the-shelf pre-trained Word-to-Vector (WV) embeddings (Mikolov et al., 2013) to get a binary tree with a depth of 25; ( ii ) tree DIS : We  use the word embeddings pre-trained for image captioning with standard MLE objective and full vocabulary, and the rest follows tree WV; ( iii ) tree RD :
Revised: We apply agglomerative clustering to off-the-shelf pre-trained Word-to-Vector (WV) embeddings (Mikolov et al., 2013) to get a binary tree with a depth of 25; ( ii ) tree DIS : We follow tree WV except that we use the word embeddings pre-trained for image captioning with standard MLE objective and full vocabulary to distill (DIS) the full-softmax model’s knowledge to the binary tree; ( iii ) tree RD :

Original: We randomly permute the leaves of tree DIS to produce a tree with no meaningful structure, referred as tree RD.
Revised: We randomly permute the leaves of tree DIS to produce a tree with no meaningful structure, referred as tree RD.

Original: We pre-train all these three models with MLE, and report the CIDEr score in Table 2.
Revised: We pre-train all these three models with MLE, and report the CIDEr score in Table 2.

Original: Among these three binary trees, tree DIS performs the best, indicating that the tree structure has impact on its performance, and a task-speciﬁc pre-trained embedding is preferable when constructing a binary tree.
Revised: Among these three binary trees, tree DIS performs the best, indicating that the tree structure has impact on its performance, and a task-speciﬁc pre-trained embedding is preferable when constructing a binary tree.

Original: As expected, comparing with the models trained using regular softmax layer (Table 2) , the performance of the models with binary-tree softmax layers drop.
Revised: As expected, comparing with the models trained using regular softmax layer (Table 2), the performance of the models with binary-tree softmax layers drop.

Original: However, the Multi-Adds softmax operations needed for generating a token is reduced by V/D ( ∼ 380 in our case) times
Revised: However, the Multi-Adds softmax operations needed for generating a token is reduced by V /D ( ∼ 380 in our case

Original:  leading to a signiﬁcant improvement in efﬁciency especially for deployment in computing resource limited scenarios at the cost of moderately degraded accuracy.
Revised: ) times, leading to a signiﬁcant improvement in efﬁciency especially for deployment in computing resource limited scenarios at the cost of moderately degraded accuracy.

Original: (b) Fine-tuning using BT-ARSM .
Revised: (b) Fine-tuning using BT-ARSM .

Original: We further ﬁne-tune the pre-trained tree DIS model, with binarytree-REINFORCE (BT-RF), binary-tree-Self-Critic (BT-SC) and binary-tree-ARSM (BT-ARSM) respectively
Revised: We further ﬁne-tune the pre-trained tree DIS model, with binarytree-REINFORCE (BT-RF), binary-tree-Self-Critic (BT-SC), and binary-tree-ARSM (BT-ARSM) respectively.

Original: Table 2 shows that BT-ARSM signiﬁcantly outperforms the other two, which can be explained by the considerable variance reduction of BT-ARSM as is shown in Fig 3b (lower).
Revised: Table 2 shows that BT- ARSM signiﬁcantly outperforms the other two, which can be explained by the considerable variance reduction of BT-ARSM as is shown in Fig 3b (lower).

Original: Notably, the performance of BT-ARSM is superior to vanilla softmax model trained with MLE even though it has been injected with strong inductive bias via binary-tree softmax to reduce its generation cost.
Revised: Notably, the performance of BT-ARSM is superior to vanilla softmax model trained with MLE even though it has been injected with strong inductive bias via binary-tree softmax to reduce its generation cost.

Original: Adaptiveness of ARS-K and BT-ARSM As shown in Fig.
Revised: Adaptiveness of ARS-K and BT-ARSM As shown in Fig.

Original: 3 and Fig.
Revised: 3 and Fig.

Original: 4 (in Appendix A), our proposed methods can adaptively choose the number of correlated MC rollouts in four aspects: ( i ) (adapt across samples) in Fig.
Revised: 4 (in Appendix A), our proposed methods can adaptively choose the number of correlated MC rollouts in four aspects: ( i ) (adapt across samples) in Fig.

Original: 3d, we show the 2-D density estimation for the numbers of rollouts and the CIDEr scores of different samples during the later stage ofthe training.
Revised: 3d, we show the 2-D density estimation for the numbers of rollouts and the CIDEr scores of different samples during the later stage of training.

Original: We observe a statistically signiﬁcant negative correlation ( p < 0 . 05 ) between the numbers of rollouts and CIDEr scores, indicating that our algorithms can adaptively generate more rollouts for harder samples (lower CIDEr scores) and less rollouts for easier ones (higher CIDEr scores); ( ii ) (adapt across iterations) as shown in Fig.
Revised: We observe a statistically signiﬁcant negative correlation ( p < 0 . 05 ) between the numbers of rollouts and CIDEr scores, indicating that our algorithms can adaptively generate more rollouts for harder samples (lower CIDEr scores) and less rollouts for easier ones (higher CIDEr scores); ( ii ) (adapt across iterations) as shown in Fig.

Original: 3c, during the training, the number of correlated MC rollouts decreases as the model improves and converges; ( iii ) (adapt across sentence positions) as shown in Figs. 4a,4b,4d,4e,
Revised: 3c, during the training, the number of correlated MC rollouts decreases as the model improves and converges; ( iii ) (adapt across sentence positions) as shown in Figs. 4a, 4b, 4d, and 4e, more MC

Original: [Table]
Revised: [Table]

Original: more MC rollouts appear at the timesteps in the middle range of the generated sequence, as they are associated with higher uncertainty; ( iv ) (adapt across depths) as shown in Figs.
Revised: rollouts appear at the timesteps in the middle range of the generated sequence, as they are associated with higher uncertainty; ( iv ) (adapt across depths) as shown in Figs.

Original: 4c, 4f, for binary-tree softmax, the top layers (close to the root) of the tree are associated with more MC rollouts, since they are more uncertain about what to predict.
Revised: 4c and 4f, for binary-tree softmax, the top layers (close to the root) of the tree are associated with more MC rollouts, since they are more uncertain about what to predict.

Original: More details are provided in Appendix A.
Revised: More details are provided in Appendix A.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we demonstrate the adaptation of ARSM policy gradient estimator, utilizing token-level rewards of correlated Monte Carlo  rollouts, to optimize contextual categorical sequence generation model.
Revised: In this paper, we demonstrate the adaptation of ARSM policy gradient estimator, utilizing tokenlevel rewards of correlated Monte Carlo (MC) rollouts, to optimize contextual categorical sequence generation model.

Original: We apply the gradient estimators based on this idea to both the regular softmax model and binary-tree softmax model.
Revised: We apply the gradient estimators based on this idea to both the regular softmax model and binary-tree softmax model.

Original: The binary-tree softmax model has low cost for generating categorical tokens and hence is suited for computation-limited scenarios.
Revised: The binary-tree softmax model has low cost for generating categorical tokens and hence is suited for computation-limited scenarios.

Original: We conduct empirical study on two challenging tasks: neural program synthesis and image captioning.
Revised: We conduct empirical study on two challenging tasks: neural program synthesis and image captioning.

Original: Our observations verify that fewer and fewer correlated Monte Carlo rollouts are conducted as the model becomes increasingly more certain during training.
Revised: Our observations verify that fewer and fewer correlated MC rollouts are conducted as the model becomes increasingly more certain during training.

Original: In addition, we show with correlated MC rollouts serving as baselines for each other, our methods show signiﬁcant reduction of gradient variance and consistently outperform related baselines.
Revised: In addition, we show with correlated MC rollouts serving as baselines for each other, our methods show signiﬁcant reduction of gradient variance and consistently outperform related baselines.

Original: We note that in a cold-start setting where we start from a complete random policy, it is still challenging to make our methods work efﬁciently as the number of pseudo actions may be too large if V is large.
Revised: We note that in a cold-start setting where we start from a complete random policy, it is still challenging to make our methods work efﬁciently as the number of pseudo actions may be too large if V is large.

Original: We consider it as future work to adapt our methods to this more challenging setting, where, to our best knowledge, little work has been done except for Ding & Soricut (2017); d’Autume et al. 
Revised: We consider it as future work to adapt our methods to this more challenging setting, where, to our best knowledge, little work has been done except for Ding & Soricut (2017) and d’Autume et al. A CKNOWLEDGMENTS

Revised: This research was supported in part by the U

Revised: S. National Science Foundation under Grant IIS-1812699.

Revised: The authors acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp

Revised: GPU used for this research, and the computational support of Texas Advanced Computing Center

Original: A A DAPTIVENESS OF ARS-K/BT-ARSM
Revised: A A DAPTIVENESS OF ARS-K/BT-ARSM

Original: [Figure]
Revised: [Figure]

Original: Adaptiveness across samples: in Figure 3d, we show the 2-D density estimation for the numbers of rollouts and the CIDEr scores for different samples during the rear part ofthe training.
Revised: Adaptiveness across samples: In Fig. 3d, we show the 2-D density estimation for the numbers of rollouts and the CIDEr scores for different samples during the late stage of training.

Original: We observe a statistically signiﬁcant negative correlation ( p < 0 . 05 ) between the number of rollouts and CIDEr scores, indicating that our algorithms can adaptively generate more rollouts for harder samples (lower
Revised: We observe a statistically signiﬁcant negative correlation ( p < 0 . 05 ) between the number of rollouts and CIDEr scores, indicating that our algorithms can adaptively generate more rollouts for harder samples (lower

Original: CIDEr scores) and less rollouts for easier samples (higher CIDEr scores).
Revised: CIDEr scores) and less rollouts for easier samples (higher CIDEr scores).

Original: Adaptiveness across time steps: seen in the Figure 4a, 4b, 4d, and 4e, it is reasonable that there will be more MC rollouts at the middle time step of the generated sequence, since the initial words and end words are more easily to be learned due to their cardinality, and ARSM model will be more conﬁdent about the choices for these words, leading to their small MC rollouts.
Revised: Adaptiveness across time steps: As shown in Figs. 4a, 4b, 4d, and 4e, it is reasonable that there will be more MC rollouts at the middle time step of the generated sequence, since the initial words and end words are more easily to be learned due to their cardinality, and ARSM model will be more conﬁdent about the choices for these words, leading to fewer MC rollouts.

Original: Adaptiveness across depths: for binary softmax model, since words are represented as leaf nodes in the tree, the ﬁrst depth should be the most uncertain to promise the reachability to enough words, and as the depth going deep, the uncertainty should be small corresponding to small MC rollouts.
Revised: Adaptiveness across depths: For binary softmax model, the terms in the vocabulary are represented as leaf nodes in the tree. As shown in Figs.

Original:  ASRM successfully captured the adaptiveness of MC rollouts across depths, seen from Figure 4c and 4f.
Revised: 4c and 4f, ASRM successfully captures the adaptiveness of MC rollouts across depths in the tree.

Revised: The top layers (close to the root) of the tree are associated with more MC rollouts, since they are more uncertain about what to predict.

Original: Adaptiveness across iterations: from Figure 1b and 3c, as the training process goes, the number of  MC rollouts is getting smaller, which is an adaptive modiﬁcation of ARSM model when the model parameters are converging and getting more certain.
Revised: Adaptiveness across iterations: In Figs. 1b and 3c, we observe that, during training, the number of correlated MC rollouts decreases as the model improves and converges.

Original: B D ETAILED FORMULATIONS B. ARS/M GRADIENT ESTIMATORS
Revised: B D ETAILED FORMULATIONS B. ARS/M GRADIENT ESTIMATORS

Original: Applying the ARSM estimator in (5) to the expected reward shown in (2) [Equation]
Revised: Applying the ARSM estimator in (5) to the expected reward shown in (2), we have ∇ φ tv ER =

Original: we have
Revised: E z 1: t − 1 ∼ p θ ( · | x ) E π t ∼ Dir ( 1 V ) [ g ARSM ( π t ) v ] , g ARSM ( π t ) v := 1 V (cid:80) Vj =1 g ARS ( π t , j ) v , g ARS ( π t , j ) v := (cid:2) r

Original: [Equation]
Revised: [Equation]

Original: where zm (cid:11) j t := argmin i ∈{ 1 ,
Revised: z

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: e − φ ti . Thus we can approximate the gradient ∇ θ ER as
Revised: (cid:11)

Original: [Equation]
Revised: [Equation]

Original:  where π
Revised: ) , where z m (cid:11) j t := argmin i ∈{ 1 ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: is an approximation of r
Revised: } π m (cid:11) j ti e − φ ti . Thus we can approximate the gradient ∇ θ ER as

Original: [Equation]
Revised: [Equation]

Original: z
Revised: g

Original: [Equation]
Revised: [Equation]

Original:  , z [Equation]
Revised: (9) where π 1 , . . , π T iid ∼ Dir ( 1 V ) ; ˆ r ( z 1: t − 1 , z m (cid:11) j t | x , y ) is an approximation of r ( z 1: t − 1 , z m (cid:11) j t

Original: [Equation]
Revised: [Equation]

Original: [ r ( z | x , y )] , which can be estimated with r ( z 1: t − 1 , z m (cid:11) j t , ˜ z t +1: T | x , y ) , where ˜ z t +1: T ∼ p θ ( · | x , z 1: t − 1 , z m (cid:11) j t ) is an MC rollout.
Revised: [ r ( z | x , y )] , which can be estimated with r ( z 1: t − 1 , z m (cid:11) j t , ˜ z t +1: T | x , y ) , where ˜ z t +1: T ∼ p θ ( · | x , z 1: t − 1 , z m (cid:11) j t ) is an MC rollout.

Original: B.
Revised: B.

Revised: [Equation]

Original: BT ARSM GRADIENT ESTIMATORS
Revised: BT-ARSM GRADIENT ESTIMATORS

Original: [Equation]
Revised: [Equation]

Original: where r ( z 1: t − 1 , b t (1: l ) | x , y ) :=
Revised: where r ( z 1: t − 1 , b t (1: l ) | x , y ) :=

Original: E b t ( l +1: Dzt ) ,z ( t +1): T ∼ p θ ( · | x ,z 1: t − 1 ,b t (1: l ) )
Revised: E b t ( l +1: Dzt ) ,z ( [Equation] 1): T ∼ p θ ( · | x ,z [Equation]

Revised: 1 ,b t (1: l ) )

Original: [ r ( z | x , y )] .
Revised: [ r ( z | x , y )] .

Original: Thus we have ∇ θ ER = (cid:80)
Revised: Thus we have ∇ θ ER = (cid:80)

Original: Tt =1 (cid:80) D zt l =1 ∇ φ t,bt (1: l − 1) ER ∇ θ φ t,b t (1: l − 1) , which can be approximated with ˆ ∇ θ ER = (cid:80)
Revised: Tt =1 (cid:80) D zt l =1 ∇ φ t,bt (1: l − 1) ER ∇ θ φ t,b t (1: l − 1) , which can be approximated with ˆ ∇ θ ER = (cid:80)

Original: Tt =1 (cid:80) D zt l =
Revised: Tt =1 (cid:80) D zt l =

Original: 1 ˆ g ARM ( π tl )(1 − 2 π tl ) , where ˆ g ARM ( π tl ) approximates g ARM ( π tl ) shown in (10) via MC integration; note if given π tl ∼ Unif (0 , 1) , b ( sudo ) tl=
Revised: 1 ˆ g ARM ( π tl )(1 − 2 π tl ) , where ˆ g ARM ( π tl ) approximates g ARM ( π tl ) shown in (10) via MC integration; note if given π tl ∼ Unif (0 , 1) , b ( sudo ) tl

Original:  b ( true ) tl , then g ARM ( π tl )
Revised: = b ( true ) tl , then g ARM ( π tl )

Original: = 0 .
Revised: = 0 .

Original: C A LGORITHMS
Revised: C A LGORITHMS

Original: Efﬁcient Pseudo-Action Computation:  to compute  pseudo actions for all V vocabularies, we do not have to take the minimum over the whole V dimensional vector for V times.
Revised: Efﬁcient Pseudo-Action Computation: To implement ARS-K, we need to compute the corresponding pseudo actions for each reference category j in a reference set J .

Original: Instead, by taking advantage of the correlation among the swapping operations of V vocabularies, we propose the following efﬁcient algorithm to compute pseudo actions.

Original: We ﬁrst explain the notations and the motivations, then present the algorithm in Algorithm 1.

Original: Purpose: given π , φ ,  compute z m (cid:11) j = argmin { i =1 ,...,V } ln π m (cid:11) j i − φ
Revised: In other words, given π , φ , we need to compute z m (cid:11) j = argmin { i =1 ,...,V } ln π m (cid:11) j i − φ

Original: i , for all m ∈ { 1 , . .
Revised: i , for all m ∈ { 1 , . .

Original: , V } , j ∈ 
Revised: , V } , j ∈ J .

Original:  J , where J is  the reference category set. Notation:
Revised: A naive implementation would involve taking the minimum over V -dimensional vectors for V · | J | times, which is computationally expensive when V is large. In the following, we take advantage of the correlation among pseudo actions and propose an efﬁcient algorithm which only involves taking the minimum over V -dimensional vectors | J | times. We ﬁrst explain the notations and the basic ideas, and then present the algorithm in Algorithm 1.

Original: Let o i,j
Revised: Let o i,j

Original: = ln π
Revised: = ln π

Original: i − φ j .
Revised: i − φ j .

Original: Denote m 1 , m 2 as the indexes for the top 2 smallest  o i,i for i = 1 : V respectively. Let ID ( o i,j )  function return the second index of o i,j , which is the actual index of o i,j in updated ln π m (cid:11) j − φ vector, since we will only exchange the π values.
Revised: Denote m 1 , m 2 as the indexes for the top 2 smallest in { o i,i } i =1: V respectively (this is the only place where we need to take the minimum over whole V -dimensional vectors). Let ID ( o i,j ) denote the function returning the second index of o i,j , which means

Original: In other words, ID ( o i,j ) =
Revised: ID ( o i,j ) =

Original: j.
Revised: j.

Revised: In the following, we show that it is not necessary to take minimum over whole V -dimensional vectors for each pseudo actions.

Original: Methods for computing the z m (cid:11) j efﬁciently:
Revised: We can compute z m (cid:11) j efﬁciently by observing:

Original: If m 1 / ∈ { m, j } Only need to compare the o m 1 ,m 1 with the updated two values created by doing π m (cid:11) j .The result is obvious, which is
Revised: If m 1 / ∈ { m, j } , the smallest value in ln π m (cid:11) j − φ can only be among the updated two values ( o j,m , o m,j ) and the original smallest value o m 1 ,m 1 .

Original: [Equation]
Revised: Hence, the index of the smallest value is z m (cid:11) j = ID (min(min( o j,m , o m,j ) , o m 1 ,m 1 )) .

Original: If m 1 ∈ { m, j }  o m 1  will be updated to a new value, while o m 2 ,m 2 is possibly not changed.
Revised: If m 1 ∈ { m, j } , o m 1 ,m 1 will be updated to a new value, therefore, the above equation does not hold.

Original: Fortunately, we can ignore checking whether m 2 ∈ { m, j } , by just comparing the o m 2 with the updated two values created by doing π m (cid:11) j , and ﬁnding the smallest value among them.
Revised: But, in the following, we show we can use the second smallest value instead. The index of the smallest value will be

Original: [Equation]
Revised: [Equation]

Original: Proof: given m 1 ∈ { m, j } if m 2 / ∈ { m, j } ,  o m 2 ,m 2 is the smallest value in the original set without including the two updated values.
Revised: Proof. Assume m 1 ∈ { m, j } . If m 2 / ∈ { m, j } , since o m 1 ,m 1 has been changed, o m 2 ,m 2 will be the smallest value in the vector excluding the two updated values.

Original: Hence, the result is obvious.
Revised: Hence, the smallest value will be among the three values { o j,m , o m,j , o m 2 ,m 2 } , among which we can ﬁnd the index of the smallest item.

Original: if m 2 ∈ { m, j } , then { m, j } becomes { m 1 , m 2 } .
Revised: If m 2 ∈ { m, j } , then { m, j } becomes { m 1 , m 2 } .

Original: Hence, o m 1  and o m 2 ,m 2 will be updated.
Revised: Hence, o m 1 ,m 1 and o m 2 ,m 2 will be updated.

Original: For notation simplicity, use m 1 , m 2 to represent m  1 ,  m [Equation] [Equation]
Revised: For any m / ∈ { m 1 , m 2 } , min( o m 2 ,m 1 , o m 1 ,m 2 ) ≤ o m 2 ,m 2 ≤ o m,m .

Original: Since we only change their π values, one of them will become smaller, while the other one becomes larger. Besides, the original o m 1 ,m 1 , o m 2 ,m 2 ≤ o m 2 ,m 2 . Hence, the smaller one in the updated two values will deﬁnitely smaller than o m 2 ,m 2  .[Equation]
Revised: Therefore, the smallest value will be among { o m 2 ,m 1 , o m 1 ,m 2 } .

Original: [Equation] [Equation] Then, min(min( o m 2 ,m 1 , o m 1 ,m 2 ) ,  o m 2 ,m 2 ) will ignore the outside condition o m 2 ,m 2 ,and also do not need to consider about o min 3 ,min 3 . Therefore, whenever o m 2 ,m 2 is present or not, we can use
Revised: The index of the smallest value will be ID (min( o m 2 ,m 1 , o m 1 ,m 2 )) , which is equivalent to ID (min(min( o j,m , o m,j ) , o m 2 ,m 2 )) .

Original: [Equation] Algorithm input : Batched π and φ , Ref-Cat Set J , ID Function output: Pseudo-Action Matrix P ;
Revised: output

Original: Computeo m 1 , m1 , o m[-2 , m 2 = Top
-]
Revised: Compute m m[-2 , m 2 = Top
-]

Original: (ln π − φ ) ; for j ∈ J, m ∈ { 1 , . .
Revised: ; for j ∈ J, m ∈ { 1 , . .

Original: , V } (in parallel) do
Revised: , V } (in parallel) do

Original: Compute o j,m =
Revised: Compute o j,m =

Original: ln π j − φ m
Revised: ln π j − φ m

Original: Compute o m,j = ln π m − φ j end for
Revised: Compute o m,j = ln π m − φ j end for

Original: Initialize P with size ( | J | , V ) for j ∈ J, m ∈ { 1 , . .
Revised: Initialize P with size ( | J | , V ) for j ∈ J, m ∈ { 1 , . .

Original: , V } (in parallel by using index matrix) do if m
Revised: , V } (in parallel by using index matrix) do if m

Original: 1 ∈ { j, m } then
Revised: 1 ∈ { j, m } then

Original: P [ j, m ] = ID (min(min( o j,m , o m,j ) , o m 1 ,m 1 )) else
Revised: P [ j, m ] = ID (min(min( o j,m , o m,j ) , o m 1 ,m 1 )) else

Original: P [ j, m ] = ID (min(min( o j,m , o m,j ) , o m 2 ,m 2 )) end if end for
Revised: P [ j, m ] = ID (min(min( o j,m , o m,j ) , o m 2 ,m 2 )) end if end for

Original: Algorithm input : MLE pre-trained policy parameter θ , number of reference category K , main trajectory sample type mt , pseudo trajectory sample type pt output : Fine-tuned policy parameter θ while not converged do
Revised: Algorithm input : MLE pre-trained policy parameter θ , number of reference category K , main trajectory sample type mt , pseudo trajectory sample type pt output : Fine-tuned policy parameter θ while not converged do

Original: Receive random sample x , y ;
Revised: Receive random sample x , y ;

Original: First, we sample a main trajectory ( z 1 , . . .
Revised: First, we sample a main trajectory ( z 1 , . . .

Original: , z T ) : if mt = ’greedily sample’ then for t = 1 : T , let z t = argmin i ∈{ 1 ,...,V } ( − φ ti ) , where φ t =
Revised: , z T ) : if mt = ’greedily sample’ then for t = 1 : T , let z t = argmin i ∈{ 1 ,...,V } ( − φ ti ) , where φ t =

Original: T θ ( z 1: t − 1 , x ) ; else for t = 1 : T , let z t = argmin i ∈{ 1 ,...,V } (ln π ti − φ ti ) , where π t ∼ Dirichlet ( 1 V ) (or let π ti = − ln( Unif (0 , 1)) ), and φ t =
Revised: T θ ( z 1: t − 1 , x ) ; else for t = 1 : T , let z t = argmin i ∈{ 1 ,...,V } (ln π ti − φ ti ) , where π t ∼ Dirichlet ( 1 V ) (or let π ti = − ln( Unif (0 , 1)) ), and φ t =

Original: T θ ( z 1: t − 1 , x ) ; end if
Revised: T θ ( z 1: t − 1 , x ) ; end if

Original: Second, we compute pseudo actions: for t = 1 : T do
Revised: Second, we compute pseudo actions: for t = 1 : T do

Original: Let π t ∼ Dirichlet ( 1 V ) (or let π ti = − ln( Unif (0 , 1)) for i = 1 , . .
Revised: Let π t ∼ Dirichlet ( 1 V ) (or let π ti = − ln( Unif (0 , 1)) for i = 1 , . .

Original: , V and then normalize them to have a unit norm);
Revised: , V and then normalize them to have a unit norm);

Original: Let j 1 , ..., j K be K reference categories randomly sampled from { 1 , . .
Revised: Let j 1 , ..., j K be K reference categories randomly sampled from { 1 , . .

Original: , V } without replacement; for k = 1 , . .
Revised: , V } without replacement; for k = 1 , . .

Original: , K , v = 1 , . .
Revised: , K , v = 1 , . .

Original: , V (in parallel) do
Revised: , V (in parallel) do

Original: Let z v (cid:11) j k t := argmin i ∈{ 1 ,...,V } (ln π v (cid:11)
Revised: Let z v (cid:11) j k t := argmin i ∈{ 1 ,...,V } (ln π v (cid:11)

Original: j k ti − φ ti ) as the ( v, k ) th pseudo action; end for
Revised: j k ti − φ ti ) as the ( v, k ) th pseudo action; end for

Original: Let S t = unique
Revised: Let S t = unique

Original: ( { z v (cid:11) j t } v,j ) which means S t is the set of all unique values in { z v (cid:11) j t } v,j .
Revised: ( { z v (cid:11) j t } v,j ) which means S t is the set of all unique values in { z v (cid:11) j t } v,j .

Original: Denote the cardinality of S t as | S t | , where 1 ≤ | S t | ≤ V ; end for
Revised: Denote the cardinality of S t as | S t | , where 1 ≤ | S t | ≤ V ; end for

Original: Third, we complete sentences and evaluate the reward for the unique set of pseudo actions: for t = 1 : T (in parallel) do if | S t | = 1 then continue end if for ˜ z ts ∈ S t (in parallel) do if t < T then if pt = ’greedily sample’ then greedily sample z st +1: T ∼ p θ ( z t +1: T | z 1: t − 1 , ˜
Revised: Third, we complete sentences and evaluate the reward for the unique set of pseudo actions: for t = 1 : T (in parallel) do if | S t | = 1 then continue end if for ˜ z ts ∈ S t (in parallel) do if t < T then if pt = ’greedily sample’ then greedily sample z st +1: T ∼ p θ ( z t +1: T | z 1: t − 1 , ˜

Original: z
Revised: z

Original: ts , x ) else randomly sample z st +1: T ∼ p θ ( z t +1: T | z 1: t − 1 , ˜ z ts , x ) end if end if end for for v = 1 : V, k = 1 : K (in parallel) do
Revised: ts , x ) else randomly sample z st +1: T ∼ p θ ( z t +1: T | z 1: t − 1 , ˜ z ts , x ) end if end if end for for v = 1 : V, k = 1 : K (in parallel) do

Original: Let f ( z v (cid:11) j k t ) =
Revised: Let f ( z v (cid:11) j k t ) =

Original: r ( z 1: t − 1 , ˜ z
Revised: r ( z 1: t − 1 , ˜ z

Original: ts , z st +1: T | x , y ) if z v (cid:11) j k t = ˜
Revised: ts , z st +1: T | x , y ) if z v (cid:11) j k t = ˜

Original: z ts ; end for end for
Revised: z ts ; end for end for

Original: Finally, we compute the ARSM gradients and update parameters: for t = 1 : T, k = 1 : K (in parallel) do
Revised: Finally, we compute the ARSM gradients and update parameters: for t = 1 : T, k = 1 : K (in parallel) do

Original: Let ¯ f tk = 1 V (cid:80)
Revised: Let ¯ f tk = 1 V (cid:80)

Original: Vv =1 f
Revised: Vv =1 f

Original: ( z v (cid:11) j k t ) ; for v = 1 : V (in parallel) do
Revised: ( z v (cid:11) j k t ) ; for v = 1 : V (in parallel) do

Original: Let g tk,v = 1 K (cid:0) f
Revised: Let g tk,v = 1 K (cid:0) f

Original: ( z v (cid:11) j k t ) − ¯ f tk (cid:1) (1 − V π tj k ) , where g tk,v is the v th component of g tk ; end for end for for t = 1 : T (in parallel) do θ
Revised: ( z v (cid:11) j k t ) − ¯ f tk (cid:1) (1 − V π tj k ) , where g tk,v is the v th component of g tk ; end for end for for t = 1 : T (in parallel) do θ

Original: = θ + η θ ∇ θ φ t g t , where g t =
Revised: = θ + η θ ∇ θ φ t g t , where g t =

Original: (cid:80)
Revised: (cid:80)

Original: k g tk with step-size
Revised: k g tk with step-size

Original: η θ end for end while input : MLE pre-trained policy parameter θ , binary code to word mapping ν output : Fine-tuned policy parameter θ ; while not converged do
Revised: η θ end for end while input : MLE pre-trained policy parameter θ , binary code to word mapping ν output : Fine-tuned policy parameter θ ; while not converged do

Original: Receive random sample x , y ;
Revised: Receive random sample x , y ;

Original: First, we sample a main trajectory ( z 1 , . . .
Revised: First, we sample a main trajectory ( z 1 , . . .

Original: , z T ) : for t = 1 : T do for d = 1 : D do φ t,b t (1: d − 1) =
Revised: , z T ) : for t = 1 : T do for d = 1 : D do φ t,b t (1: d − 1) =

Original: T θ ( z 1: t − 1 , x ) ν ( b t (1: d − 1) )
Revised: T θ ( z 1: t − 1 , x ) ν ( b t (1: d − 1) )

Original: Sample π td ∼ Uniform (0 , 1) ;
Revised: Sample π td ∼ Uniform (0 , 1) ;

Original: Let b td = 1 [ π td <σ ( φ t,bt (1: d − 1) )] ; end for z t = ν
Revised: Let b td = 1 [ π td <σ ( φ t,bt (1: d − 1) )] ; end for z t = ν

Original: ( b t (1: D ) ) end for
Revised: ( b t (1: D ) ) end for

Original: Second, we compute pseudo actions: for t = 1 : T (in parallel) do for d = 1 : D do
Revised: Second, we compute pseudo actions: for t = 1 : T (in parallel) do for d = 1 : D do

Original: Let b (1) td
Revised: Let b (1) td

Original: = b td ;
Revised: = b td ;

Original: Let b (2) td = 1 [ π td >σ ( − φ t,bt (1: d − 1) )] ; if b (1) td (cid:54) =
Revised: Let b (2) td = 1 [ π td >σ ( − φ t,bt (1: d − 1) )] ; if b (1) td (cid:54) =

Original: b (2) td then
Revised: b (2) td then

Original: If d < D , sample b ( j ) t ( d +1: D ) ∼ p θ ( b t ( d +1: D ) | z 1: t − 1 , b t, 1: d − 1 , b ( j ) td , x ) , j = 1 , 2 ;
Revised: If d < D , sample b ( j ) t ( d +1: D ) ∼ p θ ( b t ( d +1: D ) | z 1: t − 1 , b t, 1: d − 1 , b ( j ) td , x ) , j = 1 , 2 ;

Original: Let z ( j ) td
Revised: Let z ( j ) td

Original: = ν ( b ( j ) t (1: D ) ) , j = 1 , 2 ; end if end for
Revised: = ν ( b ( j ) t (1: D ) ) , j = 1 , 2 ; end if end for

Original: Let S t = unique
Revised: Let S t = unique

Original: ( { z ( j ) td } d,j ) which means S t is the set of all unique values in { z ( j ) td } d,j .
Revised: ( { z ( j ) td } d,j ) which means S t is the set of all unique values in { z ( j ) td } d,j .

Original: Denote the cardinality of S t as | S t | , where D ≤ | S t | ≤ 2 ∗ D. end for
Revised: Denote the cardinality of S t as | S t | , where D ≤ | S t | ≤ 2 ∗ D. end for

Original: Third, we complete sentences and evaluate the rewards for the unique set of pseudo actions: for t = 1 : T (in parallel) do for ˜ z
Revised: Third, we complete sentences and evaluate the rewards for the unique set of pseudo actions: for t = 1 : T (in parallel) do for ˜ z

Original: ts ∈ S t (in parallel) do
Revised: ts ∈ S t (in parallel) do

Original: If t < T , sample z st +1: T ∼ p θ ( z t +1: T | z 1: t − 1 , ˜
Revised: If t < T , sample z st +1: T ∼ p θ ( z t +1: T | z 1: t − 1 , ˜

Original: z
Revised: z

Original: ts , x ) ; end for for d = 1 : D, j = 1 : 2 (in parallel) do
Revised: ts , x ) ; end for for d = 1 : D, j = 1 : 2 (in parallel) do

Original: Let f ( j ) td
Revised: Let f ( j ) td

Original: = r
Revised: = r

Original: ( z 1: t − 1 , ˜ z
Revised: ( z 1: t − 1 , ˜ z

Original: ts , z st +1: T | x , y ) if z ( j ) td = ˜
Revised: ts , z st +1: T | x , y ) if z ( j ) td = ˜

Original: z ts ; end for
Revised: z ts ; end for

Original: We compute the ARSM gradients and update parameters: for d = 1 : D (in parallel) do if b (1) td (cid:54) =
Revised: We compute the ARSM gradients and update parameters: for d = 1 : D (in parallel) do if b (1) td (cid:54) =

Original: b (2) tl then
Revised: b (2) tl then

Original: Let g t,b t (1: d − 1)
Revised: Let g t,b t (1: d − 1)

Original: = 12 ( f (1) td − f (2) td )(1 − 2 π td ) ;
Revised: = 12 ( f (1) td − f (2) td )(1 − 2 π td ) ;

Original: θ update =
Revised: θ update =

Original: η θ ∇ θ φ t,b t (1: d − 1) g t,b t (1: d − 1) , with step-size η θ end if θ = θ + θ update end for end for end while
Revised: η θ ∇ θ φ t,b t (1: d − 1) g t,b t (1: d − 1) , with step-size η θ end if θ = θ + θ update end for end for end while

Original: D E XPERIMENTAL S ETUP D ETAILS D. I MAGE C APTIONING
Revised: D E XPERIMENTAL S ETUP D ETAILS D. I MAGE C APTIONING

Original: Image captioning maps an image x to a sentence y =
Revised: Image captioning maps an image x to a sentence y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y T ) that summarizes the image information.
Revised: , y T ) that summarizes the image information.

Original: It has become a standard task to compare different RL methods using policy gradient.
Revised: It has become a standard task to compare different RL methods using policy gradient.

Original: A popular evaluation metric for this task is the CIDEr score (Vedantam et al., 2015), which measures the similarity between the generated caption y and some reference ones.
Revised: A popular evaluation metric for this task is the CIDEr score (Vedantam et al., 2015), which measures the similarity between the generated caption y and some reference ones.

Original: (Rennie et al., 2017; Anderson et al., 2018; Xu et al., 2015).
Revised: (Rennie et al., 2017; Anderson et al., 2018; Xu et al., 2015).

Original: We conduct our experiments on the MS COCO dataset (Lin et al., 2014) which consists of 123,287 images.
Revised: We conduct our experiments on the MS COCO dataset (Lin et al., 2014) that consists of 123,287 images.

Original: Each image has at least ﬁve captions.
Revised: Each image has at least ﬁve captions.

Original: We use the standard data split from Karpathy & Fei-Fei (2015), with 113,287 training, 5000 validation, and 5000 testing images.
Revised: We use the standard data split from Karpathy & Fei-Fei (2015), with 113,287 training, 5000 validation, and 5000 testing images.

Original: The vocabulary size V is 9488 and the max caption length T is 16.
Revised: The vocabulary size V is 9488 and the max caption length T is 16.

Original: For the model architecture, we employ a Fully-Connected (FC) model without attention (Rennie et al., 2017).
Revised: For the model architecture, we employ a Fully-Connected (FC) model without attention (Rennie et al., 2017).

Original: Image features are extracted from a pre-trained ResNet (He et al., 2016).
Revised: Image features are extracted from a pre-trained ResNet (He et al., 2016).

Original: Our implementation is based on Luo et al.
Revised: Our implementation is based on Luo et al.

Original: We pre-train a model with MLE until convergence and use it for initialization.
Revised: We pre-train a model with MLE until convergence and use it for initialization.

Original: The CIDEr scores between the generated captions and references are used as reward.
Revised: The CIDEr scores between the generated captions and references are used as rewards.

Original: E Q UALITATIVE RESULTS E. P 
Revised: E Q UALITATIVE RESULTS E. P SEUDO SENTENCES PRODUCED BY ARS-K ALGORITHM .

Original: SEUDO SENTENCES PRODUCED BY ARS-K ALGORITHM

Original: Main sentence: a man riding a motorcycle with a dog.
Revised: Main sentence: a man riding a motorcycle with a dog.

Original: Pseudo sentence 1: a man riding a motorcycle with hay on it.
Revised: Pseudo sentence 1: a man riding a motorcycle with hay on it.

Original: Pseudo sentence 2: a man riding a motorcycle with pack of sheep.
Revised: Pseudo sentence 2: a man riding a motorcycle with pack of sheep.

Original: Main sentence: a group of people ﬂying kites in a ﬁeld.
Revised: Main sentence: a group of people ﬂying kites in a ﬁeld.

Original: Pseudo sentence 1: a group of teenagers standing in a ﬁeld ﬂying kites.
Revised: Pseudo sentence 1: a group of teenagers standing in a ﬁeld ﬂying kites.

Original: Main sentence: a man and woman are standing in a of a table.
Revised: Main sentence: a man and woman are standing in a of a table.

Original: Pseudo sentence 1: a man and woman are standing in an market.
Revised: Pseudo sentence 1: a man and woman are standing in an market.

Original: Pseudo sentence 2: a man and woman are standing in the street.
Revised: Pseudo sentence 2: a man and woman are standing in the street.

Original: Pseudo sentence 3: a man and woman are standing in to a tent.
Revised: Pseudo sentence 3: a man and woman are standing in to a tent.

Original: Pseudo sentence 4: a man and woman are standing in front of a table.
Revised: Pseudo sentence 4: a man and woman are standing in front of a table.

Original: Main sentence: a city that has a large white building on it.
Revised: Main sentence: a city that has a large white building on it.

Original: Pseudo sentence 1: a city with a red trafﬁc and a large building.
Revised: Pseudo sentence 1: a city with a red trafﬁc and a large building.

Original: Pseudo sentence 2: a city intersection with a trafﬁc light and a street sign.
Revised: Pseudo sentence 2: a city intersection with a trafﬁc light and a street sign.

Original: Pseudo sentence 3: a city bus is driving down the street.
Revised: Pseudo sentence 3: a city bus is driving down the street.

Original: Pseudo sentence4: a city street with a city street with cars parked on it.
Revised: Pseudo sentence a city street with a city street with cars parked on it.

Original: Pseudo sentence 5: a city road with a trafﬁc light and astreet sign.
Revised: Pseudo sentence 5: a city road with a trafﬁc light and a

Revised: street

Revised: sign