Paper:
We derive an unbiased estimator for expectations over discrete random variables based on sampling without replacement, which reduces variance as it avoids duplicate samples. We show that our estimator can be derived as the RaoBlackwellization of three different estimators. Combining our estimator with RE-INFORCE, we obtain a policy gradient estimator and we reduce its variance using a built-in control variate which is obtained without additional model evaluations. The resulting estimator is closely related to other gradient estimators. Experiments with a toy problem, a categorical Variational Auto-Encoder and a structured prediction problem show that our estimator is the only estimator that is consistently among the best estimators in both high and low entropy settings.

Review:
Edited after rebuttal:
I'm satisfied with the clarifications provided by the authors about where they see this work being applicable. I think they have correctly identified a setting that is of interest to a broad audience and where their approach is more attractive than the alternatives. I now recommend accepting this paper.
The authors develop a generic method for estimating expectations under discrete distributions based on sampling without replacement and apply it to the task of estimating gradients for training models with discrete latent variables. The proposed estimator is unbiased and the authors prove it has lower variance than the naive Monte Carlo estimator.
I weakly recommend rejecting the paper. While the derivations and theorems presented in the paper are correct, the experiments are sensible and support the claims made, my impression is that the range of applications where the proposed method is a better alternative to the existing algorithms is extremely narrow. I would be willing to raise my rating to Weak Accept if the authors convince me that their method is useful more broadly than I think it is.
The method derivation is generally well-written and easy to follow. However, the account of the sum-and-sample estimator is somewhat misleading. As is, it suggests that it is natural to sum over k-1 elements and only draw one sample, while in fact choosing how many elements to sum over (I'll call this number S in my review) and how many samples to draw from the remaining set is the crucial step needed to make this algorithm work well. This is discussed by Fearnhead and Clifford [1] in the context of resampling in particle filters and I believe is also addressed by Liu et al. (2019). I understand that the authors are guiding the presentation towards a version of sum-and-sample they can compare with in theory and in experiments, but in the process they're doing a disservice to the reader. I think the authors should state the importance of choosing S well early on and then either compare with methods that attempt to set S optimally throughout experiments, or clearly explain that they restrict their discussion to a particular setting where S is difficult to optimize and explain why.
There are also additional baselines that should be considered. This includes methods such as stratified or systematic sampling, which are often discussed within the context of resampling [2]. Another strong but somewhat obscure baseline is given by Duffield et al. [3]. While it is possible that none of those methods are applicable in the experimental setting chosen in the paper, the authors should be very clear about how broadly they claim superiority of their method.
On the topic of experimental evaluation, the authors seem to have focused on the setting where sampling is done using stochastic beam search, presumably because in such a setting some of the strong baselines mentioned above, including sum-and-sample with the optimal choice of S, are not applicable. If that's true, I would like to see a detailed discussion about what it is about the problem setting that makes alternative baselines inapplicable. My current impression is that the proposed method is only worth considering when used with a stochastic beam search and I don't think such settings are particularly common. This is my main complaint about the paper.
On the subject of applicability, the experiments only test the proposed method in very low k settings, where the exponential algorithm for computing the leave-one-out ratio can be applied. While the appendix mentions that there is a faster alternative based on numerical integration, the claim that it can be efficiently integrated is not substantiated anywhere.
The plots could use some improvement. It took me a lot of squinting at Figures 2 and 3 to figure out what's going on, since the lines overlap too much - I couldn't even find the Unordered line in Figure 2 at first. Also I'm confused about why lines for sum-and-sample in Figure 1 extend beyond 8 evaluations, and VIMCO does not. Also figure captions could be a little more self-contained.
Finally, I would recommend stating Theorem 1 in more concrete terms, such as "estimator A has lower variance than estimator B".
[1] P. Fearnhead and P. Clifford. On-line inference for hidden Markov models via particle filters. Journal of the Royal Statistical Society, 65:887–899, 2003.
[2] R. Douc, O. Cappé, and E. Moulines. Comparison of resampling schemes for particle filtering. In ISPA, 2005.
[3] N. Duffield, C. Lund, and M. Thorup. Priority sampling for estimation of arbitrary subset sums. Journal of the ACM, 54, 2007.