Original: B UY 4 REINFORCE SAMPLES , GET A BASELINE FOR FREE !
Revised: B UY 4 REINFORCE S AMPLES , G ET A B ASELINE FOR F REE !

Original: Anonymous authors Paper under double-blind review
Revised: w.w.m.kool@uva.nl van c vanhoof@uva

Original: A BSTRACT
Revised: A BSTRACT

Original: REINFORCE can be used to train models in structured prediction settings to directly optimize the test-time objective.
Revised: REINFORCE can be used to train models in structured prediction settings to directly optimize the test-time objective.

Original: However, the common case of sampling one prediction per datapoint (input) is data-inefﬁcient.
Revised: However, the common case of sampling one prediction per datapoint (input) is data-inefﬁcient.

Original: We show that by drawing multiple samples (predictions) per datapoint, we can learn with signiﬁcantly less data, as we freely obtain a REINFORCE baseline to reduce variance.
Revised: We show that by drawing multiple samples (predictions) per datapoint, we can learn with signiﬁcantly less data, as we freely obtain a REINFORCE baseline to reduce variance.

Original: Additionally we derive a REINFORCE estimator with baseline, based on sampling without replacement.
Revised: Additionally we derive a REINFORCE estimator with baseline, based on sampling without replacement.

Original: Combined with a recent technique to sample sequences without replacement using Stochastic Beam Search, this improves the training procedure for a sequence model that predicts the solution to the Travelling Salesman Problem.
Revised: Combined with a recent technique to sample sequences without replacement using Stochastic Beam Search, this improves the training procedure for a sequence model that predicts the solution to the Travelling Salesman Problem.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: REINFORCE (Williams, 1992) is a well known policy optimization algorithm that learns directly from experience.
Revised: REINFORCE (Williams, 1992) is a well known policy optimization algorithm that learns directly from experience.

Original: Variants of it have been used to train models for a wide range of structured prediction tasks, such as Neural Machine Translation (Ranzato et al., 2016;
Revised: Variants of it have been used to train models for a wide range of structured prediction tasks, such as Neural Machine Translation (Ranzato et al., 2016;

Original: Bahdanau et al., 2017), Image
Revised: Bahdanau et al., 2017), Image

Original: Captioning (Vinyals et al., 2015b) and predicting solutions (tours) for the Travelling Salesman Problem (TSP) (Bello et al., 2016; Kool et al., 2019a).
Revised: Captioning (Vinyals et al., 2015b) and predicting solutions (tours) for the Travelling Salesman Problem (TSP) (Bello et al., 2016; Kool et al., 2019a).

Original: As opposed to maximum likelihood (supervised) learning, the appeal of using REINFORCE for structured prediction is that it directly optimizes the test-time performance.
Revised: As opposed to maximum likelihood (supervised) learning, the appeal of using REINFORCE for structured prediction is that it directly optimizes the test-time performance.

Original: When using REINFORCE, often for each datapoint (e.g. a sentence, image or TSP instance) only a single sample/prediction (e.g. a translation, caption or tour) is used to construct a gradient estimate.
Revised: When using REINFORCE, often for each datapoint (e.g. a sentence, image or TSP instance) only a single sample/prediction (e.g. a translation, caption or tour) is used to construct a gradient estimate.

Original: From a classic Reinforcement Learning (RL) point of view, this makes sense, as we may not be able to evaluate multiple sampled actions for a state (datapoint).
Revised: From a classic Reinforcement Learning (RL) point of view, this makes sense, as we may not be able to evaluate multiple sampled actions for a state (datapoint).

Original: However, from a data point of view, this is inefﬁcient if we can actually evaluate multiple samples, such as in a structured prediction setting.
Revised: However, from a data point of view, this is inefﬁcient if we can actually evaluate multiple samples, such as in a structured prediction setting.

Revised: Reinforcement Learning with multiple samples/predictions for a single datapoint has been used before (e.g. Shen et al.

Original: By using multiple samples, we obtain counterfactual information which can be used to construct a (local, for a single datapoint) REINFORCE baseline, increasing the speed of learning through the achieved variance reduction.
Revised: ; He et al. , but we use the samples as counterfactual information by constructing a (local, for a single datapoint) REINFORCE baseline. A similar idea was applied for variational inference by Mnih & Rezende (2016).

Original: Many structured prediction tasks can be formulated in terms of sequence modelling, which is the focus of this paper.
Revised: Many structured prediction tasks can be formulated in terms of sequence modelling, which is the focus of this paper.

Original: In most sequence modelling tasks, the objective is a deterministic function of the predicted sequence.
Revised: In most sequence modelling tasks, the objective is a deterministic function of the predicted sequence.

Original: As a result, duplicate sampled sequences are uninformative and therefore do not improve the quality of the gradient estimate.
Revised: As a result, duplicate sampled sequences are uninformative and therefore do not improve the quality of the gradient estimate.

Original: To solve this problem, we propose to use sampling without replacement to construct a better gradient estimate.
Revised: To solve this problem, we propose to use sampling without replacement to construct a better gradient estimate.

Original: This is inspired by recent work by Kool et al.
Revised: This is inspired by recent work by Kool et al.

Original: (2019b), who introduce Stochastic Beam Search as a method to sample sequences without replacement, and use this to construct a (normalized) importance-weighted estimator for (sentence level) BLEU score.
Revised: (2019b), who introduce Stochastic Beam Search as a method to sample sequences without replacement, and use this to construct a (normalized) importance-weighted estimator for (sentence level) BLEU score.

Original: We extend this idea to estimate policy gradients using REINFORCE, and we show how to use the same set of samples (without replacement) to construct a baseline.
Revised: We extend this idea to estimate policy gradients using REINFORCE, and we show how to use the same set of samples (without replacement) to construct a baseline.

Original: This way we can leverage sampling without replacement to improve training of sequence models.
Revised: This way we can leverage sampling without replacement to improve training of sequence models.

Original: In our experiment, we consider the TSP and show that using REINFORCE with multiple samples is beneﬁcial compared to single sample REINFORCE, both computationally and in terms of data-efﬁciency.
Revised: In our experiment, we consider the TSP and show that using REINFORCE with multiple samples is beneﬁcial compared to single sample REINFORCE, both computationally and in terms of data-efﬁciency.

Original: Additionally, for a sample size of 4 − 8 samples per datapoint, sampling without replacement results in slightly faster learning.
Revised: Additionally, for a sample size of 4 − 8 samples per datapoint, sampling without replacement results in slightly faster learning.

Original:  B ACKGROUND 2. REINFORCE
Revised:  B ACKGROUND 2. REINFORCE

Original: The REINFORCE estimator (Williams, 1992) allows to estimate gradients of the expectation E y ∼ p θ ( y )
Revised: The REINFORCE estimator (Williams, 1992) allows to estimate gradients of the expectation E y ∼ p θ [Equation] y )

Original: [ f ( y )] by the relation:
Revised: [ f ( y )] by the relation:

Original: [Equation]
Revised: [Equation]

Original: If we also have a context, or datapoint, x (such as a source sentence), we may write p θ ( y | x ) and f ( y, x ) , but in this paper, we leave dependence on x implicit.
Revised: If we also have a context, or datapoint, x (such as a source sentence), we may write p θ ( y | x ) and f ( y, x ) , but in this paper, we leave dependence on x implicit.

Original: Extension of the derived estimators to a minibatch of datapoints x is straightforward.
Revised: Extension of the derived estimators to a minibatch of datapoints x is straightforward.

Original: Typically, we estimate the expectation using samples y 1 , ..., y k and we may reduce variance of the estimator by using a baseline B i that is independent of the sample y i (but may depend on the other samples y j , j (cid:54) =
Revised: Typically, we estimate the expectation using samples y 1 , ..., y k and we may reduce variance of the estimator by using a baseline B i that is independent of the sample y i (but may depend on the other samples y j , j (cid:54) =

Original: i ):
Revised: i ):

Original: [Equation]
Revised: [Equation]

Original: In practice, often a single sample y is used (per datapoint x , as we already have a batch of datapoints) to compute the estimate, e.g. k
Revised: In practice, often a single sample y is used (per datapoint x , as we already have a batch of datapoints) to compute the estimate, e.g. k

Original: = 1 , but in this paper we consider k > 1 .
Revised: = 1 , but in this paper we consider k > 1 .

Original: 2. T HE G UMBEL -T OP - k TRICK AND S TOCHASTIC B EAM S EARCH
Revised: 2. T HE G UMBEL -T OP - k TRICK AND S TOCHASTIC B EAM S EARCH

Original: In this paper, we consider a parametric distribution over discrete structures (sequences).
Revised: In this paper, we consider a parametric distribution over discrete structures (sequences).

Original: Enumerating all n possible sequences as y 1 , ..., y n , we indicate with y i
Revised: Enumerating all n possible sequences as y 1 , ..., y n , we indicate with y i

Original: the i -th possible outcome, which has log-probability φ i
Revised: the i -th possible outcome, which has log-probability φ i

Original: = log p θ ( y i ) deﬁned by the model.
Revised: = log p θ ( y i ) deﬁned by the model.

Original: We can use the Gumbel-Max trick (Gumbel, 1954; Maddison et al., 2014) to sample y according to this distribution as follows: let
Revised: We can use the Gumbel-Max trick (Gumbel, 1954; Maddison et al., 2014) to sample y according to this distribution as follows: let

Original: G i ∼ Gumbel (a standard Gumbel distribution) for i = 1 , ..., n i.i.d., and let y =
Revised: G i ∼ Gumbel (a standard Gumbel distribution) for i = 1 , ..., n i.i.d., and let y =

Original: y i ∗ , where i ∗ = arg max i { φ i + G i } .
Revised: y i ∗ , where i ∗ = arg max i { φ i + G i } .

Original: Then P ( y = y i ) =
Revised: Then P ( y = y i ) =

Original: p θ ( y i ) .
Revised: p θ ( y i ) .

Original: For a proof we refer to Maddison et al.
Revised: For a proof we refer to Maddison et al.

Original: In a slight abuse of notation, we write G φ i
Revised: In a slight abuse of notation, we write G φ i

Original: =
Revised: =

Original: φ i + G i , and we call G φ
Revised: φ i + G i , and we call G φ

Original: i the (Gumbel-) perturbed log-probability of y i .
Revised: i the (Gumbel-) perturbed log-probability of y i .

Original: The Gumbel-Max trick can be extended to the Gumbel-Top- k trick (Kool et al., 2019b) to draw an ordered sample without replacement , by taking the top k largest perturbed log-probabilities (instead of just one, the argmax).
Revised: The Gumbel-Max trick can be extended to the Gumbel-Top- k trick (Kool et al., 2019b) to draw an ordered sample without replacement , by taking the top k largest perturbed log-probabilities (instead of just one, the argmax).

Original: The result is equivalent to sequential sampling without replacement, where after an element y is sampled, it is removed from the domain and the remaining probabilities are renormalized.
Revised: The result is equivalent to sequential sampling without replacement, where after an element y is sampled, it is removed from the domain and the remaining probabilities are renormalized.

Original: The Gumbel-Top- k trick is equivalent to Weighted Reservoir Sampling (Efraimidis & Spirakis, 2006), as was noted by Vieira (2014).
Revised: The Gumbel-Top- k trick is equivalent to Weighted Reservoir Sampling (Efraimidis & Spirakis, 2006), as was noted by Vieira (2014).

Revised: The ordered sample is also known as a partial ranking according to the Plackett-Luce model (Plackett, 1975; Luce, 1959).

Original: For a sequence model with exponentially large domain, naive application of the Gumbel-Top- k trick is infeasible, but an equivalent result can be obtained using Stochastic Beam Search (Kool et al., 2019b).
Revised: For a sequence model with exponentially large domain, naive application of the Gumbel-Top- k trick is infeasible, but an equivalent result can be obtained using Stochastic Beam Search (Kool et al., 2019b).

Original: This modiﬁcation of beam search expands the k partial sequences with maximum (Gumbel) perturbed log-probability, effectively replacing the standard top k operation by sampling without replacement.
Revised: This modiﬁcation of beam search expands the k partial sequences with maximum (Gumbel) perturbed log-probability, effectively replacing the standard top k operation by sampling without replacement.

Original: The resulting top k completed sequences are a sample without replacement from the sequence model, by the equivalence to the Gumbel-Top- k trick.
Revised: The resulting top k completed sequences are a sample without replacement from the sequence model, by the equivalence to the Gumbel-Top- k trick.

Original: For details we refer to
Revised: For details we refer to

Original: Kool et al. (2019b).
Revised: Kool et al. (2019b).

Original: 2. S TATISTICAL ESTIMATION WITHOUT REPLACEMENT
Revised: 2. S TATISTICAL ESTIMATION WITHOUT REPLACEMENT

Original: For many applications we need to estimate the expectation of a function f ( y ) , where y is the realization of a variable with a discrete probability distribution p θ ( y ) .
Revised: For many applications we need to estimate the expectation of a function f ( y ) , where y is the realization of a variable with a discrete probability distribution p θ ( y ) .

Original: When using Monte Carlo (MC) sampling (with replacement), we write y i to indicate the i -th sample in a set of samples.
Revised: When using Monte Carlo (MC) sampling (with replacement), we write y i to indicate the i -th sample in a set of samples.

Original: In contrast, when sampling without replacement we ﬁnd it convenient to write y
Revised: In contrast, when sampling without replacement we ﬁnd it convenient to write y

Original: i (with superscript i ) to refer to the i -th possible value in the domain, so (like we did in Section 2.2) we can enumerate the domain with n possible values as y 1 , ..., y n .
Revised: i (with superscript i ) to refer to the i -th possible value in the domain, so (like we did in Section 2.2) we can enumerate the domain with n possible values as y 1 , ..., y n .

Original: This notation allows us to write out the expectation of f ( y ) :
Revised: This notation allows us to write out the expectation of f ( y ) :

Original: [Equation]
Revised: [Equation]

Original: Using MC sampling with replacement, we estimate equation 3 using k samples y 1 , ..., y k :
Revised: Using MC sampling with replacement, we estimate equation 3 using k samples y 1 , ..., y k :

Original: [Equation]
Revised: [Equation]

Original: When sampling without replacement using the Gumbel-Top- k trick (Section 2.2) we write S as the set of k largest indices of G φ
Revised: When sampling without replacement using the Gumbel-Top- k trick (Section 2.2) we write S as the set of k largest indices of G φ

Original: i (i.e. S = arg top k { G φ i : i ∈ { 1 , ..., n }} ), so the sample (of size k ) without replacement is { y i : i ∈ S } .
Revised: i (i.e. S = arg top k { G φ i : i ∈ { 1 , ..., n }} ), so the sample (of size k ) without replacement is { y i : i ∈ S } .

Original: We can use the sample S with the estimator derived by
Revised: We can use the sample S with the estimator derived by

Original: Vieira (2017), based on priority sampling (Dufﬁeld et al., 2007).
Revised: Vieira (2017), based on priority sampling (Dufﬁeld et al., 2007).

Original: This means that, to correct for the effects of sampling without replacement, we include importance weights p θ ( y i ) q θ ,κ ( y i ) .
Revised: This means that, to correct for the effects of sampling without replacement, we include importance weights p θ ( y i ) q θ ,κ ( y i ) .

Original: Here κ is the ( k + 1) -th largest value of { G φ i : i ∈ { 1 , ..., n }} , i.e. the ( k + 1) -th largest Gumbel perturbed log-probability, and q θ ,a ( y i ) =
Revised: Here κ is the ( k + 1) -th largest value of { G φ i : i ∈ { 1 , ..., n }} , i.e. the ( k + 1) -th largest Gumbel perturbed log-probability, and q θ ,a ( y i ) =

Original: P( G
Revised: P

Original: [Equation]
Revised: [Equation]

Original:  i > a ) = 1 − exp(exp( φ i − a )) is the probability that the perturbed log-probability of y i exceeds a .
Revised: G φ i > a ) = 1 − exp( − exp( φ i − a )) is the probability that the perturbed log-probability of y i exceeds a .

Original: Then we can use the following estimator:
Revised: Then we can use the following estimator:

Original: [Equation]
Revised: [Equation]

Original: This estimator is unbiased, and we include a copy of the proof by Kool et al.
Revised: This estimator is unbiased, and we include a copy of the proof by Kool et al.

Original: (2019b) (adapted from the proofs by Dufﬁeld et al.
Revised: (2019b) (adapted from the proofs by Dufﬁeld et al.

Original: and Vieira (2017)) in Appendix A, as this introduces notation and is the basis for the proof in Appendix C.
Revised: and Vieira (2017)) in Appendix A, as this introduces notation and is the basis for the proof in Appendix C.

Original: Intuition behind this estimator comes from the related threshold sampling scenario, where instead of ﬁxing the sample size k , we ﬁx the threshold a and deﬁne a variably sized sample
Revised: Intuition behind this estimator comes from the related threshold sampling scenario, where instead of ﬁxing the sample size k , we ﬁx the threshold a and deﬁne a variably sized sample

Original: S =
Revised: S =

Original: { i ∈ { 1 , ..., n } : G φ i > a } .
Revised: { i ∈ { 1 , ..., n } : G φ i > a } .

Original: With threshold sampling, each element y i in the domain is sampled independently with probability P ( G φ i > a ) =
Revised: With threshold sampling, each element y i in the domain is sampled independently with probability P ( G φ i > a ) =

Original: q θ ,a ( y i ) , and p θ ( y i ) q θ ,a ( y i ) is a standard importance weight.
Revised: q θ ,a ( y i ) , and p θ ( y i ) q θ ,a ( y i ) is a standard importance weight.

Original: As it turns out, instead of having a ﬁxed threshold a , we can ﬁx the sample size k and use κ as empirical threshold (as i ∈ S if G φ i > κ ), and still obtain an unbiased estimator (Dufﬁeld et al., 2007; Vieira, 2017).
Revised: As it turns out, instead of having a ﬁxed threshold a , we can ﬁx the sample size k and use κ as empirical threshold (as i ∈ S if G φ i > κ ), and still obtain an unbiased estimator (Dufﬁeld et al., 2007; Vieira, 2017).

Original: As was shown by Kool et al.
Revised: As was shown by Kool et al.

Original: (2019b), in practice it is preferred to normalize the importance weights to reduce variance.
Revised: (2019b), in practice it is preferred to normalize the importance weights to reduce variance.

Original: This means that we compute the normalization W ( S ) =
Revised: This means that we compute the normalization W ( S ) =

Original: (cid:80) i ∈
Revised: (cid:80) i ∈

Original: S p θ ( y i ) q θ ,κ ( y i ) and obtain the following (biased) estimator:
Revised: S p θ ( y i ) q θ ,κ ( y i ) and obtain the following (biased) estimator:

Original: [Equation]
Revised: [Equation]

Original:  REINFORCE WITH MULTIPLE SAMPLES
Revised:  REINFORCE WITH MULTIPLE SAMPLES

Original: Typically REINFORCE is applied with a single sample y per datapoint x
Revised: Typically REINFORCE is applied with a single sample y per datapoint x

Original: (e.g. one translation per source sentence, or, in our experiment, a single tour per TSP instance).
Revised: (e.g. one translation per source sentence, or, in our experiment, a single tour per TSP instance).

Original: In some cases, it may be preferred to take multiple samples y per datapoint x as this requires less data.
Revised: In some cases, it may be preferred to take multiple samples y per datapoint x as this requires less data.

Original: Taking multiple samples also gives us counterfactual information which can be used to construct a strong (local) baseline.
Revised: Taking multiple samples also gives us counterfactual information which can be used to construct a strong (local) baseline.

Original: Additionally, we obtain computational beneﬁts, as for encoder-decoder models we can obtain multiple samples using only a single pass through the encoder.
Revised: Additionally, we obtain computational beneﬁts, as for encoder-decoder models we can obtain multiple samples using only a single pass through the encoder.

Original: 3. REINFORCE WITH REPLACEMENT
Revised: 3. REINFORCE WITH REPLACEMENT

Original: With replacement, we can use the estimator in equation 2, where we can construct a baseline B i for the i -th term based on the other samples j (cid:54)
Revised: With replacement, we can use the estimator in equation 2, where we can construct a baseline B i for the i -th term based on the other samples j (cid:54)

Original: =
Revised: =

Original: i : B i = 1 k
Revised: i : B i = 1 k

Original: [Equation]
Revised: [Equation]

Original: 1 (cid:80) j (cid:54) =
Revised: 1 (cid:80) j (cid:54) =

Original: i f ( y j ) .
Revised: i f ( y j ) .

Original: We obtain the following
Revised: We obtain the following

Original: REINFORCE estimator based on drawing k samples y 1 , ..., y k with replacement:
Revised: REINFORCE estimator based on drawing k samples y 1 , ..., y k with replacement:

Original: [Equation]
Revised: [Equation]

Original: The form in equation 8 is convenient for implementation as it allows to compute a ﬁxed ‘baseline’
Revised: The form in equation 8 is convenient for implementation as it allows to compute a ﬁxed ‘baseline’

Original: B = 1 k (cid:80) kj =1 f ( y j ) once and correct for the bias (as B depends on y i ) by normalizing using 1 k −instead of 1 k .
Revised: B = 1 k (cid:80) kj =1 f ( y j ) once and correct for the bias (as B depends on y i ) by normalizing using 1 k −instead of 1 k .

Original: For details and a proof of unbiasedness we refer to Appendix C.
Revised: For details and a proof of unbiasedness we refer to Appendix C.

Original: 3. REINFORCE WITHOUT REPLACEMENT
Revised: 3. REINFORCE WITHOUT REPLACEMENT

Original: The basic REINFORCE without replacement estimator follows from combining equation 1 with equation 5 for an unbiased estimator:
Revised: The basic REINFORCE without replacement estimator follows from combining equation 1 with equation 5 for an unbiased estimator:

Original: [Equation]
Revised: [Equation]

Original: Similar to equation 6, we can compute a lower variance but biased variant by normalizing the importance weights using the normalization W ( S ) =
Revised: Similar to equation 6, we can compute a lower variance but biased variant by normalizing the importance weights using the normalization W ( S ) =

Original: (cid:80)
Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: When sampling without replacement, the individual samples are dependent , and therefore we cannot simply deﬁne a baseline based on the other samples as we did in Section 3.1.
Revised: When sampling without replacement, the individual samples are dependent , and therefore we cannot simply deﬁne a baseline based on the other samples as we did in Section 3.1.

Original: However, similar to the ‘baseline’ 1 k (cid:80) kj =1 f ( y j ) in equation 8, we can deﬁne an estimate of E y ∼ p θ ( y )
Revised: However, similar to the ‘baseline’ 1 k (cid:80) kj =1 f ( y j ) in equation 8, we can deﬁne an estimate of E y ∼ p θ ( y )

Original: [ f ( y )]
Revised: [ f ( y )]

Original: based on the complete sample S (without replacement), using equation 5: B ( S )
Revised: based on the complete sample S (without replacement), using equation 5: B ( S )

Original: = (cid:80)
Revised: = (cid:80)

Original: j ∈ S p θ ( y j ) q θ ,κ ( y j ) f
Revised: j ∈ S p θ ( y j ) q θ ,κ ( y j ) f

Original: ( y j ) .
Revised: ( y j ) .

Original: Using this baseline introduces a bias that we cannot simply correct for by a constant term (as we did in equation 8), as the importance weights depend on y i .
Revised: Using this baseline introduces a bias that we cannot simply correct for by a constant term (as we did in equation 8), as the importance weights depend on y i .

Original: Instead, we weight the individual terms by
Revised: Instead, we weight the individual terms by

Original: [Equation]
Revised: [Equation]

Original: This estimator is unbiased and we give the full proof in Appendix C.
Revised: This estimator is unbiased and we give the full proof in Appendix C.

Original: For the normalized version, we use the normalization W ( S ) =
Revised: For the normalized version, we use the normalization W ( S ) =

Original: (cid:80) i ∈
Revised: (cid:80) i ∈

Original: S p θ ( y i ) q θ ,κ ( y i ) for the baseline, and
Revised: S p θ ( y i ) q θ ,κ ( y i ) for the baseline, and

Original: W i ( S ) =
Revised: W i ( S ) =

Original: W ( S ) − p θ ( y i )
Revised: W ( S ) − p θ ( y i )

Original: q θ ,κ ( y i )
Revised: q θ ,κ ( y i )

Original: + p θ ( y i ) to normalize the outer terms:
Revised: + p θ ( y i ) to normalize the outer terms:

Original: [Equation]
Revised: [Equation]

Original: It seems odd to normalize the terms in the outer sum by 1 W i ( S ) instead of 1 W ( S ) , but this estimator can be considered the (normalized) without-replacement equivalent of equation 8 where we normalize by 1 k − 1 instead of 1 k .
Revised: It seems odd to normalize the terms in the outer sum by 1 W i ( S ) instead of 1 W ( S ) , but this estimator can be considered the (normalized) without-replacement equivalent of equation 8 where we normalize by 1 k − 1 instead of 1 k .

Original: It can be derived by rewriting equation 10 to a form similar to equation 7 (as this reveals the actual ‘baseline’), then applying the normalization of the importance-weights for the outer sum and the baseline, and then rewriting it once again to the form similar to equation 8 to obtain equation 11.
Revised: It can be derived by rewriting equation 10 to a form similar to equation 7 (as this reveals the actual ‘baseline’), then applying the normalization of the importance-weights for the outer sum and the baseline, and then rewriting it once again to the form similar to equation 8 to obtain equation 11.

Original: This derivation is given in full in Appendix C.1.
Revised: This derivation is given in full in Appendix C.1.

Original: The estimator in equation 11 is convenient to implement as the (normalized) ‘baseline’ B ( S ) W ( S ) only has to be computed once.
Revised: The estimator in equation 11 is convenient to implement as the (normalized) ‘baseline’ B ( S ) W ( S ) only has to be computed once.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: We consider the task of predicting the solution for instances of the Travelling Salesman Problem (TSP) (Vinyals et al., 2015a;
Revised: We consider the task of predicting the solution for instances of the Travelling Salesman Problem (TSP) (Vinyals et al., 2015a;

Original: Bello et al., 2016; Kool et al., 2019a).
Revised: Bello et al., 2016; Kool et al., 2019a).

Original: The problem is to ﬁnd the order in which to visit locations (speciﬁed by their x, y coordinates) to minimize total travelling distance.
Revised: The problem is to ﬁnd the order in which to visit locations (speciﬁed by their x, y coordinates) to minimize total travelling distance.

Original: A policy is trained using REINFORCE to minimize the expected length of a tour (sequence of locations) predicted by the model.
Revised: A policy is trained using REINFORCE to minimize the expected length of a tour (sequence of locations) predicted by the model.

Original: The Attention Model by Kool et al. (2019a) is a sequence model that considers each instance as a fully connected graph of nodes which are processed by an encoder.
Revised: The Attention Model by Kool et al. (2019a) is a sequence model that considers each instance as a fully connected graph of nodes which are processed by an encoder.

Original: The decoder then produces the tour as a sequence of nodes to visit, one node at a time, where it autoregressively uses as input the node visited in the previous step.
Revised: The decoder then produces the tour as a sequence of nodes to visit, one node at a time, where it autoregressively uses as input the node visited in the previous step.

Original: [Figure]
Revised: [Figure]

Original: We use the source code by Kool et al.
Revised: We use the source code by Kool et al.

Original: (2019a)
Revised: (2019a)

Original: 1 to reproduce their TSP experiment with 20 nodes (as larger instances diminish the beneﬁt of sampling without replacement).
Revised: 1 to reproduce their TSP experiment with 20 nodes (as larger instances diminish the beneﬁt of sampling without replacement).

Original: We implement REINFORCE estimators based on multiple samples, either sampled with replacement (WR) or without replacement (WOR) using Stochastic Beam Search (Kool et al., 2019b).
Revised: We implement REINFORCE estimators based on multiple samples, either sampled with replacement (WR) or without replacement (WOR) using Stochastic Beam Search (Kool et al., 2019b).

Original: We compare the following four estimators
Revised: We compare the following four estimators

Original: : • Single sample with a batch baseline .
Revised: : • Single sample with a batch baseline .

Original: Here we compute the standard REINFORCE estimator (equation 2) with a single sample ( k = 1 ).
Revised: Here we compute the standard REINFORCE estimator (equation 2) with a single sample ( k = 1 ).

Original: We use a batch of 512 instances (datapoints) and as baseline we take the average of the tour lengths in the batch , hence each instances uses the same baseline.
Revised: We use a batch of 512 instances (datapoints) and as baseline we take the average of the tour lengths in the batch , hence each instances uses the same baseline.

Original: This is implemented as using the exponential moving average baseline by (Kool et al., 2019a) with β = 0 .
Revised: This is implemented as using the exponential moving average baseline by (Kool et al., 2019a) with β = 0 .

Original: • Single sample with a greedy rollout baseline , and batch size 512.
Revised: • Single sample with a greedy rollout baseline , and batch size 512.

Original: As baseline, we use a greedy rollout : for each instance x we take the length of the tour that is obtained by greedily selecting the next location according to an earlier (frozen) version of the model.
Revised: As baseline, we use a greedy rollout : for each instance x we take the length of the tour that is obtained by greedily selecting the next location according to an earlier (frozen) version of the model.

Original: This baseline, similar to self-critical training (Rennie et al., 2017), corresponds to the best result found by Kool et al. (2019a), superior to using an exponential moving average or learned value function.
Revised: This baseline, similar to self-critical training (Rennie et al., 2017), corresponds to the best result found by Kool et al. (2019a), superior to using an exponential moving average or learned value function.

Original: However, the greedy rollout requires an additional forward pass through the model.
Revised: However, the greedy rollout requires an additional forward pass through the model.

Original: • Multiple samples with replacement (WR) with a local baseline .
Revised: • Multiple samples with replacement (WR) with a local baseline .

Original: Here we compute the estimator in equation 8 based on k = 4 , 8 samples.
Revised: Here we compute the estimator in equation 8 based on k = 4 , 8 samples.

Original: We use a batch size of 512 k , so the total number of samples is the same.
Revised: We use a batch size of 512 k , so the total number of samples is the same.

Original: The baseline is local as it is different for each datapoint, but it does not require additional model evaluations like the greedy rollout.
Revised: The baseline is local as it is different for each datapoint, but it does not require additional model evaluations like the greedy rollout.

Original: • Multiple samples without replacement (WOR) with a local baseline .
Revised: • Multiple samples without replacement (WOR) with a local baseline .

Original: Here we use the (biased) normalized without replacement estimator in equation 11 with k = 4 , 8 samples and batch size 512 k .
Revised: Here we use the (biased) normalized without replacement estimator in equation 11 with k = 4 , 8 samples and batch size 512 k .

Original: Samples are drawn without replacement using Stochastic Beam Search (Kool et al., 2019b).
Revised: Samples are drawn without replacement using Stochastic Beam Search (Kool et al., 2019b).

Original: For fair comparison, we do not take a ( k +1)
Revised: For fair comparison, we do not take a ( k +1)

Original: -th sample to compute κ , but sacriﬁce the k -th sample and compute the summation in equation 11 with the remaining k − 1 (3 or 7) samples.
Revised: -th sample to compute κ , but sacriﬁce the k -th sample and compute the summation in equation 11 with the remaining k − 1 (3 or 7) samples.

Original: Note that a single sample with a local baseline is not possible, which is why we use the batch baseline.
Revised: Note that a single sample with a local baseline is not possible, which is why we use the batch baseline.

Original: The model architecture and training hyperparameters (except batch size) are as in the paper by
Revised: The model architecture and training hyperparameters (except batch size) are as in the paper by

Original: Kool et al. (2019a).
Revised: Kool et al. (2019a).

Original: We present the results in terms of the validation set (not used for additional tuning) optimality gap during training in Figure 1, using k = 4 (top row) and k = 8 (bottom row).
Revised: We present the results in terms of the validation set (not used for additional tuning) optimality gap during training in Figure 1, using k = 4 (top row) and k = 8 (bottom row).

Original: We found diminishing returns for larger k .
Revised: We found diminishing returns for larger k .

Original: The left column presents the results in terms of the number of gradient update steps (minibatches).
Revised: The left column presents the results in terms of the number of gradient update steps (minibatches).

Original: We see that sampling without replacement performs on par ( k = 8 ) or slightly better than using the strong but computationally expensive greedy rollout baseline or using multiple samples with replacement.
Revised: We see that sampling without replacement performs on par ( k = 8 ) or slightly better than using the strong but computationally expensive greedy rollout baseline or using multiple samples with replacement.

Original: The standard batch baseline performs signiﬁcantly worse.
Revised: The standard batch baseline performs signiﬁcantly worse.

Original: The estimators based on multiple samples do not lose (much) ﬁnal performance, while using signiﬁcantly less instances.
Revised: The estimators based on multiple samples do not lose (much) ﬁnal performance, while using signiﬁcantly less instances.

Original: In the right column, where results are presented in terms of the number of instances, this effectiveness is conﬁrmed, and we observe that sampling without replacement is preferred to sampling with replacement.
Revised: In the right column, where results are presented in terms of the number of instances, this effectiveness is conﬁrmed, and we observe that sampling without replacement is preferred to sampling with replacement.

Original: The difference is small, but there is also not much room for improvement as results are close to optimal.
Revised: The difference is small, but there is also not much room for improvement as results are close to optimal.

Original: The beneﬁt of learning with less data may be small if data is easily generated (as in our setting), but there is also a signiﬁcant computational beneﬁt as we need signiﬁcantly fewer encoder evaluations.
Revised: The beneﬁt of learning with less data may be small if data is easily generated (as in our setting), but there is also a signiﬁcant computational beneﬁt as we need signiﬁcantly fewer encoder evaluations.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this paper, we have derived REINFORCE estimators based on drawing multiple samples, with and without replacement, and evaluated the effectiveness of the proposed estimators in a structured prediction setting: the prediction of tours for the TSP.
Revised: In this paper, we have derived REINFORCE estimators based on drawing multiple samples, with and without replacement, and evaluated the effectiveness of the proposed estimators in a structured prediction setting: the prediction of tours for the TSP.

Original: The derived estimators yield results comparable to recent results using REINFORCE with a strong greedy rollout baseline, at greater data-efﬁciency and computational efﬁciency.
Revised: The derived estimators yield results comparable to recent results using REINFORCE with a strong greedy rollout baseline, at greater data-efﬁciency and computational efﬁciency.

Original: These estimators are especially well suited for structured prediction settings, where the domain is too large to compute exact gradients, but we are able to take multiple samples for the same datapoint, and the objective is a deterministic function of the sampled prediction.
Revised: These estimators are especially well suited for structured prediction settings, where the domain is too large to compute exact gradients, but we are able to take multiple samples for the same datapoint, and the objective is a deterministic function of the sampled prediction.

Original: We hope the proposed estimators have potential to be used to improve training efﬁciency in more structured prediction settings, for example in the context of Neural Machine Translation or Image Captioning, where depending on the entropy of the model, sampling without replacement may yield a beneﬁcial improvement.
Revised: We hope the proposed estimators have potential to be used to improve training efﬁciency in more structured prediction settings, for example in the context of Neural Machine Translation or Image Captioning, where depending on the entropy of the model, sampling without replacement may yield a beneﬁcial improvement.

Original: A P ROOF OF UNBIASEDNESS OF PRIORITY SAMPLING ESTIMATOR
Revised: A P ROOF OF UNBIASEDNESS OF PRIORITY SAMPLING ESTIMATOR

Original: We include here in full the proof by Kool et al.
Revised: We include here in full the proof by Kool et al.

Original: (2019b), as this introduces necessary notation and helps understanding of the proof in Appendix C.
Revised: (2019b), as this introduces necessary notation and helps understanding of the proof in Appendix C.

Original: A. P ROOF OF UNBIASEDNESS OF PRIORITY SAMPLING ESTIMATOR BY K OOL ET AL . (2019 B )
Revised: A. P ROOF OF UNBIASEDNESS OF PRIORITY SAMPLING ESTIMATOR BY K OOL ET AL . (2019 B )

Original: The following proof is adapted from the proofs by Dufﬁeld et al.
Revised: The following proof is adapted from the proofs by Dufﬁeld et al.

Original: and Vieira (2017).
Revised: and Vieira (2017).

Original: For generality of the proof, we write f ( i ) =
Revised: For generality of the proof, we write f ( i ) =

Original: f ( y i ) , p i = p θ ( y i ) and q i ( κ ) =
Revised: f ( y i ) , p i = p θ ( y i ) and q i ( κ ) =

Original: q θ ,κ ( y i ) , and we consider general keys h
Revised: q θ ,κ ( y i ) , and we consider general keys h

Original: i (not necessarily Gumbel perturbations).
Revised: i (not necessarily Gumbel perturbations).

Original: We assume we have a probability distribution over a ﬁnite domain 1 , ..., n with normalized probabilities p
Revised: We assume we have a probability distribution over a ﬁnite domain 1 , ..., n with normalized probabilities p

Original: i , e.g. (cid:80) ni
Revised: i , e.g. (cid:80) ni

Original: =1 p i = 1 .
Revised: =1 p i = 1 .

Original: For a given function f ( i ) we want to estimate the expectation
Revised: For a given function f ( i ) we want to estimate the expectation

Original: E [ f ( i )
Revised: E [ f ( i )]

Original: [Equation] = [Equation]
Revised: = n (cid:88) i = p

Original: i [Equation]
Revised: i f ( i ) .

Original: Each element i has an associated random key h
Revised: Each element i has an associated random key h

Original: i and we deﬁne q
Revised: i and we deﬁne q

Original: i ( a ) =
Revised: i ( a ) =

Original: P ( h i > a ) .
Revised: P ( h i > a ) .

Original: This way, if we know the threshold a it holds that q i ( a ) =
Revised: This way, if we know the threshold a it holds that q i ( a ) =

Original: P ( i ∈ S ) is the probability that element i is in the sample S .
Revised: P ( i ∈ S ) is the probability that element i is in the sample S .

Original: As was noted by Vieira (2017), the actual distribution of the key does not inﬂuence the unbiasedness of the estimator but does determine the effective sampling scheme.
Revised: As was noted by Vieira (2017), the actual distribution of the key does not inﬂuence the unbiasedness of the estimator but does determine the effective sampling scheme.

Original: Using the Gumbel perturbed log-probabilities as keys (e.g. h i =
Revised: Using the Gumbel perturbed log-probabilities as keys (e.g. h i =

Original: G φ i ) is equivalent to the PPSWOR scheme described by Vieira (2017).
Revised: G φ i ) is equivalent to the PPSWOR scheme described by Vieira (2017).

Original: We deﬁne shorthand notation h 1: n = { h 1 , ..., h n } , h − i
Revised: We deﬁne shorthand notation h 1: n = { h 1 , ..., h n } , h − i

Original: = { h 1 , ..., h i − 1 , h
Revised: = { h 1 , ..., h i − 1 , h

Original: i +1 , ..., h n } =
Revised: i +1 , ..., h n } =

Original: h 1: n \{ h i } .
Revised: h 1: n \{ h i } .

Original: For a given sample size k , let κ be the ( k + 1) -th largest element of h 1: n , so κ is the empirical threshold .
Revised: For a given sample size k , let κ be the ( k + 1) -th largest element of h 1: n , so κ is the empirical threshold .

Original: Let κ (cid:48)
Revised: Let κ (cid:48)

Original: i be the k -th largest element of h − i (the k -th largest of all other elements).
Revised: i be the k -th largest element of h − i (the k -th largest of all other elements).

Original: Similar to Dufﬁeld et al.
Revised: Similar to Dufﬁeld et al.

Original: we will show that every element i in our sample contributes an unbiased estimate of E [ f ( i )] , so that the total estimator is unbiased.
Revised: we will show that every element i in our sample contributes an unbiased estimate of E [ f ( i )] , so that the total estimator is unbiased.

Original: Formally, we will prove that
Revised: Formally, we will prove that

Original: E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (12) from which the result follows:
Revised: 12) from which the result follows:

Original: [Equation]
Revised: [Equation]

Original: To prove equation 12, we make use of the observation (slightly rephrased) by Dufﬁeld et al.
Revised: To prove equation 12, we make use of the observation (slightly rephrased) by Dufﬁeld et al.

Original: (2007) that conditioning on h − i , we know κ (cid:48)
Revised: (2007) that conditioning on h − i , we know κ (cid:48)

Original: i and the event i ∈ S implies that κ = κ (cid:48)
Revised: i and the event i ∈ S implies that κ = κ (cid:48)

Original: i since i will only be in the sample if h i > κ (cid:48)
Revised: i since i will only be in the sample if h i > κ (cid:48)

Original: i which means that κ (cid:48)
Revised: i which means that κ (cid:48)

Original: i is the k + 1 -th largest value of h − i ∪ { h i } = h 1: n .
Revised: i is the k + 1 -th largest value of h − i ∪ { h i } = h 1: n .

Original: The reverse is also true (if κ = κ (cid:48)
Revised: The reverse is also true (if κ = κ (cid:48)

Original: i then h i must be larger than κ (cid:48)
Revised: i then h i must be larger than κ (cid:48)

Original: i since otherwise the k + 1 -th largest
Revised: i since otherwise the k + 1 -th largest

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: q i ( κ (cid:48) i ) (cid:21) =
Revised: i ( κ (cid:48) i ) (cid:21) =

Original: E
Revised: E

Original: h − i (cid:20)
Revised: h − i (cid:20)

Original: E h
Revised: E h

Original: i (cid:20) q i ( κ (cid:48) i ) (cid:21) q i ( κ (cid:48) i ) (cid:21) =
Revised: i (cid:20) q i ( κ (cid:48) i ) (cid:21) q i ( κ (cid:48) i ) (cid:21) =

Original: E
Revised: E

Original: h − i (cid:20) q
Revised: h − i (cid:20) q

Original: i ( κ (cid:48) i ) q i ( κ (cid:48) i ) (cid:21) =
Revised: i ( κ (cid:48) i ) q i ( κ (cid:48) i ) (cid:21) =

Original: E
Revised: E

Original: h − i [1] =
Revised: h − i [1] [Equation]

Original: B REINFORCE WITH BASELINE AND REPLACEMENT
Revised: B REINFORCE WITH BASELINE AND REPLACEMENT

Original: We will now prove that the REINFORCE estimator based on multiple samples with the sample average as baseline (equation 8) is unbiased.
Revised: We will now prove that the REINFORCE estimator based on multiple samples with the sample average as baseline (equation 8) is unbiased.

Original: Let y 1: k = { y 1 , ..., y k } be the set of independent samples (with replacement) from p θ ( y ) .
Revised: Let y 1: k = { y 1 , ..., y k } be the set of independent samples (with replacement) from p θ ( y ) .

Original: First we show that using the batch mean as baseline is equivalent to using the mean of the other elements in the batch, up to a constant k − 1 k .[Equation] i
Revised: First we show that using the batch mean as baseline is equivalent to using the mean of the other elements in the batch, up to a constant k − 1 k .

Original: [Equation]
Revised: [Equation]

Original: Note that k − 1 k goes to 1 as the batch size k increases and we do not need to include it (and we can simply compute the biased mean) as it can be absorbed into the learning rate.
Revised: Note that k − 1 k goes to 1 as the batch size k increases and we do not need to include it (and we can simply compute the biased mean) as it can be absorbed into the learning rate.

Original: Since y j is independent of y i , unbiasedness follows:
Revised: Since y j is independent of y i , unbiasedness follows:

Original: [Equation]
Revised: [Equation]

Original: 

Original: 

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: C REINFORCE WITH BASELINE WITHOUT REPLACEMENT
Revised: C REINFORCE WITH BASELINE WITHOUT REPLACEMENT

Original: The proof that the REINFORCE estimator based on multiple samples without replacement with baseline (equation 10) is unbiased follows from adapting and combining the proofs in Appendix A and B. Additionally to q i ( a ) =
Revised: The proof that the REINFORCE estimator based on multiple samples without replacement with baseline (equation 10) is unbiased follows from adapting and combining the proofs in Appendix A and B. Additionally to q i ( a ) =

Original: P ( h i > a ) we deﬁne q ij ( a ) =
Revised: P ( h i > a ) we deﬁne q ij ( a ) =

Original: P ( h i > a ∩ h j > a ) =
Revised: P ( h i > a ∩ h j > a ) =

Original: P ( h i > a ) P ( h j > a ) =
Revised: P ( h i > a ) P ( h j > a ) =

Original: q i ( a ) q j ( a ) for i (cid:54) =
Revised: q i ( a ) q j ( a ) for i (cid:54) =

Original: j and q ii ( a ) =
Revised: j and q ii ( a ) =

Original: P ( h i > a ) =
Revised: P ( h i > a ) =

Original: q i ( a ) .
Revised: q i ( a ) .

Original: For convenience we deﬁne shorthand for the conditional q j | i ( a )
Revised: For convenience we deﬁne shorthand for the conditional q j | i ( a )

Original: =
Revised: =

Original: q ij ( a ) q i ( a ) , so q j | i ( a ) =
Revised: q ij ( a ) q i ( a ) , so q j | i ( a ) =

Original: q j ( a ) for j (cid:54) =
Revised: q j ( a ) for j (cid:54) =

Original: i and q i | i ( a )
Revised: i and q i | i ( a )

Original: = 1 .
Revised: = 1 .

Original: Furthermore, we deﬁne h − ij = h 1: n \ { h i , h j } and deﬁne κ (cid:48) ij ( i (cid:54) =
Revised: Furthermore, we deﬁne h − ij = h 1: n \ { h i , h j } and deﬁne κ (cid:48) ij ( i (cid:54) =

Original: j ) as the ( k − 1) -th
Revised: j ) as the ( k − 1) -th

Original: (not k -th!)
Revised: (not k -th!)

Original: largest element of h − ij , and κ (cid:48) ii
Revised: largest element of h − ij , and κ (cid:48) ii

Original: = κ (cid:48)
Revised: = κ (cid:48)

Original: i , e.g. the k -th largest element of h − i .
Revised: i , e.g. the k -th largest element of h − i .

Original: We denote with with { i, j ∈ S } =
Revised: We denote with with { i, j ∈ S } =

Original: { i ∈ S ∩ j ∈ S } the event that both i and j are in the sample, also for i = j which simply means { i ∈ S } .
Revised: { i ∈ S ∩ j ∈ S } the event that both i and j are in the sample, also for i = j which simply means { i ∈ S } .

Original: First we generalize P ( i ∈ S | h − i )
Revised: First we generalize P ( i ∈ S | h − i )

Original: =
Revised: =

Original: q i ( κ (cid:48) i ) to the pairwise conditional inclusion probability P ( i, j ∈ S | h − ij ) .
Revised: q i ( κ (cid:48) i ) to the pairwise conditional inclusion probability P ( i, j ∈ S | h − ij ) .

Original: Lemma 1.
Revised: Lemma 1.

Original: P ( i, j ∈ S | h − ij ) =
Revised: P ( i, j ∈ S | h − ij ) =

Original: q ij ( κ (cid:48) ij )
Revised: q ij ( κ (cid:48) ij )

Original: Proof.
Revised: Proof.

Original: For i =
Revised: For i =

Original: j :
Revised: j :

Original: P ( [Equation]
Revised: P ( i, j ∈ S | h − ij ) = P ( i ∈ S | h − i )

Original: [Equation]
Revised: =

Original: = q ii ( κ (cid:48) 
Revised: q i ( κ (cid:48) i ) =

Original: [Equation]

Original:  ) =q [Equation]
Revised: q ii ( κ (cid:48) ii ) =

Original:  ( κ (cid:48) ij )
Revised: q ij ( κ (cid:48) ij )

Original: For i (cid:54) =
Revised: For i (cid:54) =

Original: j : Assuming w.l.o.g.
Revised: j : Assuming w.l.o.g.

Original: h
Revised: h

Original: i < h
Revised: i < h

Original: j there are the following scenario’s:
Revised: j there are the following scenario’s:

Original: • κ (cid:48) ij < h i < h j .
Revised: • κ (cid:48) ij < h i < h j .

Original: In this case, after adding h i and h j to h − ij , κ (cid:48) ij will be the ( k + 1) -th largest element so κ = κ (cid:48) ij
Revised: In this case, after adding h i and h j to h − ij , κ (cid:48) ij will be the ( k + 1) -th largest element so κ = κ (cid:48) ij

Original: and i ∈ S and j ∈ S since h j > h i > κ = κ (cid:48) ij .
Revised: and i ∈ S and j ∈ S since h j > h i > κ = κ (cid:48) ij .

Original: • h
Revised: • h

Original: i < κ (cid:48) ij < h j or h
Revised: i < κ (cid:48) ij < h j or h

Original: i < h
Revised: i < h

Original: j < κ (cid:48) ij .
Revised: j < κ (cid:48) ij .

Original: In both cases, there are at least ( k − 1) + 1 =
Revised: In both cases, there are at least ( k − 1) + 1 =

Original: k elements higher than h
Revised: k elements higher than h

Original: i
Revised: i

Original: so i (cid:54)∈ S .
Revised: so i (cid:54)∈ S .

Original: Therefore it follows that { i, j ∈ S | u − ij } =
Revised: Therefore it follows that { i, j ∈ S | u − ij } =

Original: { h i > κ (cid:48) ij ∩ h j > κ (cid:48) ij | u − ij } and additionally this event implies κ = κ (cid:48) ij .
Revised: { h i > κ (cid:48) ij ∩ h j > κ (cid:48) ij | u − ij } and additionally this event implies κ = κ (cid:48) ij .

Original: Now the result follows:
Revised: Now the result follows:

Original: [Equation]
Revised: [Equation]

Original: Using this Lemma we can prove the following Lemma:
Revised: Using this Lemma we can prove the following Lemma:

Original: Lemma 2.
Revised: Lemma 2.

Original: [Equation]
Revised: [Equation]

Original: Note that the expectation is w.r.t.
Revised: Note that the expectation is w

Original:  the keys h 1: n which deﬁne the random variables κ and S
Revised: [Equation] r [Equation] t. the keys h 1: n which deﬁne the random variables κ and S

Original: = { i : h i > κ } .
Revised: = { i : h i > κ } .

Original: Proof.
Revised: Proof.

Revised: E h 1: n

Original: [Equation]
Revised: [Equation]

Original: Theorem 1.
Revised: Theorem 1.

Original: Let B ( S ) =
Revised: Let B ( S ) =

Original: (cid:80)
Revised: (cid:80)

Original: j ∈ S p θ ( y j ) q j ( κ ) f ( y j ) .
Revised: j ∈ S p θ ( y j ) q j ( κ ) f ( y j ) .

Original: Then the following is an unbiased estimator:
Revised: Then the following is an unbiased estimator:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First note that, when i ∈ S , we can rewrite:
Revised: First note that, when i ∈ S [Equation] we can rewrite:

Original: [Equation]
Revised: [Equation]

Original: Then the proof follows:
Revised: Then the proof follows:

Original: [Equation]
Revised: [Equation]

Original: i )   f ( y i ) · E h 1: n (cid:20) 1 { i ∈ S } q i ( κ ) (cid:21) − n (cid:88)
Revised: )   f ( y i ) · E h 1: n (cid:20) 1 { i ∈ S } q i ( κ ) (cid:21) − n (cid:88)

Original: j = p θ ( y j ) f
Revised: j = p θ ( y j ) f

Original: ( y j ) · E h 1: n (cid:20) 1 { i,j ∈ S } q ij ( κ ) (
Revised: ( y j ) · E h 1: n (cid:20) 1 { i,j ∈ S } q ij ( κ ) (

Original: cid:21) = n (cid:88) i = ∇ θ p θ ( y i )   f ( y i ) · 1 − n (cid:88)
Revised: cid:21) = n (cid:88) i = ∇ θ p θ ( y i )   f ( y i ) · 1 − n (cid:88)

Original: j = p θ ( y j ) f
Revised: j = p θ ( y j ) f

Original: ( y j ) ·
Revised: ( y j ) ·

Original:  = n (cid:88) i = ∇ θ p θ ( y i ) (cid:0) f
Revised:  = n (cid:88) i = ∇ θ p θ ( y i ) (cid:0) f

Original: ( y i ) − E y ∼ p
Revised: ( y i ) − E y ∼ p

Original: θ ( y )
Revised: θ ( y )

Original: [ f ( y )] (cid:1) =
Revised: [ f ( y )] (cid:1) =

Original: n (cid:88) i = ∇ θ p
Revised: n (cid:88) i = ∇ θ p

Original: θ ( y i )
Revised: θ ( y i )

Original: f ( y i ) − n (cid:88) i = ∇ θ p θ ( y i )
Revised: f ( y i ) − n (cid:88) i = ∇ θ p θ ( y i )

Original: E
Revised: E

Original: y ∼ p θ ( y )
Revised: y ∼ p θ ( y )

Original: [ f ( y )]
Revised: [ f ( y )]

Original: = n (cid:88) i = ∇ θ p θ ( y i ) f
Revised: = n (cid:88) i = ∇ θ p θ ( y i ) f

Original: ( y i ) − E y ∼ p
Revised: ( y i ) − E y ∼ p

Original: θ ( y )
Revised: θ ( y )

Original: [ f ( y )] · n (cid:88)
Revised: [ f ( y )] · n (cid:88)

Original: i = ∇ θ p θ ( y i ) =
Revised: i = ∇ θ p θ ( y i ) =

Original: ∇ θ n (cid:88) i = p θ ( y i )
Revised: ∇ θ n (cid:88) i = p θ ( y i )

Original: f
Revised: f

Original: ( y i ) − E y ∼ p
Revised: ( y i ) − E y ∼ p

Original: θ ( y )
Revised: θ ( y )

Original: [ f ( y )] · ∇ θ n (cid:88)
Revised: [ f ( y )] · ∇ θ n (cid:88)

Original: i = p
Revised: i = p

Original: θ ( y i )
Revised: θ ( y i )

Original: =
Revised: =

Original: ∇ θ E y ∼ p θ ( y )
Revised: ∇ θ E y ∼ p θ ( y )

Original: [ f ( y )]
Revised: [ f ( y )]

Original: − E y ∼ p θ ( y )
Revised: − E y ∼ p θ ( y )

Original: [ f ( y )] · ∇ θ= ∇ θ E y ∼ p θ ( y ) [ f ( y )] − E y ∼ p θ ( y ) [ f ( y )] ·= ∇ θ E y ∼ p θ ( y )
Revised: [ f ( y )] · ∇ θ= ∇ θ E y ∼ p θ ( y ) [ f ( y )] − E y ∼ p θ ( y ) [ f ( y )] ·= ∇ θ E y ∼ p θ ( y )

Original: [ f ( y )]
Revised: [ f ( y )]

Original: C
Revised: C. N ORMALIZED IMPORTANCE WEIGHTS By

Original: ORMALIZED IMPORTANCE WEIGHTS

Original: By rewriting ∇ θ p 
Revised: rewriting ∇ θ p θ

Original: [Equation]
Revised: [Equation]

Original: = [Equation] κ ) · ∇ θ log p θ we reveal that the estimator equation 15 is actually an importance weighted REINFORCE estimator.
Revised: log p θ we reveal that the estimator equation 15 is actually an importance weighted REINFORCE estimator.

Original: Although unbiased, in practice it is preferred to use the normalized importance weight estimator.
Revised: Although unbiased, in practice it is preferred to use the normalized importance weight estimator.

Original: We will therefore normalize both the outer weights as well as the weights used for computation of the baseline.
Revised: We will therefore normalize both the outer weights as well as the weights used for computation of the baseline.

Original: For this, write W ( S )
Revised: For this, write W ( S )

Original: =
Revised: =

Original: (cid:80) i ∈
Revised: (cid:80) i ∈

Original: S p θ ( y i )
Revised: S p θ ( y i )

Original: q i ( κ ) and scale equation 15 by 1 W ( S ) .
Revised: q i ( κ ) and scale equation 15 by 1 W ( S ) .

Original: Note that although equation 15 is written in terms of B ( S ) , the actual baseline (for sample i ) is (cid:80) j ∈ S p θ ( y j ) q j | i ( κ ) f ( y j ) (see equation 16) which should be normalized by
Revised: Note that although equation 15 is written in terms of B ( S ) , the actual baseline (for sample i ) is (cid:80) j ∈ S p θ ( y j ) q j | i ( κ ) f ( y j ) (see equation 16) which should be normalized by

Original: [Equation]
Revised: [Equation]

Original: Using equation [Equation] we can rewrite (similar to equation 13)
Revised: Using equation 16, we can rewrite (similar to equation 13)

Original: [Equation]
Revised: [Equation]

Original: Substituting this into equation 15 and normalizing the outer importance weights by W ( S ) we see that this term cancels to obtain 
Revised: Substituting this into equation 15 and normalizing the outer importance weights by W ( S ) we see that this term cancels to obtain [Equation] i

Original: [Equation]
Revised: [Equation]