Paper:
Reparameterization Gradient for Non-differentiable Models
We present a new algorithm for stochastic variational inference that targets at models with non-differentiable densities. One of the key challenges in stochastic variational inference is to come up with a low-variance estimator of the gradient of a variational objective. We tackle the challenge by generalizing the reparameterization trick, one of the most effective techniques for addressing the variance issue for differentiable models, so that the trick works for non-differentiable models as well. Our algorithm splits the space of latent variables into regions where the density of the variables is differentiable, and their boundaries where the density may fail to be differentiable. For each differentiable region, the algorithm applies the standard reparameterization trick and estimates the gradient restricted to the region. For each potentially non-differentiable boundary, it uses a form of manifold sampling and computes the direction for variational parameters that, if followed, would increase the boundary's contribution to the variational objective. The sum of all the estimates becomes the gradient estimate of our algorithm. Our estimator enjoys the reduced variance of the reparameterization gradient while remaining unbiased even for non-differentiable models. The experiments with our preliminary implementation confirm the benefit of reduced variance and unbiasedness.

Review:
EDITED AFTER REBUTTAL PERIOD:  The rebuttal addresses concerns 3 and 4. I like the plots in Figure 1' and suggest to include them in the paper, together with table 2'.  Regarding concerns 1 and 2, I disagree with the authors' rebuttal in that Gumble-softmax (and related) approaches are "orthogonal" to the paper. As noted in the rebuttal, any model with categorical variables can be rewritten in terms of if/else statements. That is exactly what I had in mind when I wrote my initial review, which is why I suggested to include an experiment involving a model with categorical latent variables.  ==============================  In this paper, a VI stochastic gradient estimator for non-differentiable models is proposed. The proposed estimator contains two terms; one resembles the reparameterization gradient and the other involves surface integrals over the non-differentiable boundary. The estimator is compared on three models involving if statements to the score function estimator as well as a naive approach that takes into account the reparameterization component only.  To the best of my knowledge, the method presented in this paper is novel. The paper is very well written and the ideas are easy to follow, although I am not familiar with the results in [3] about exchanging integration and differentiation under moving domain, so I can't really assess the correctness of the math and the proofs. They are technically sound, though.  I have some comments and questions, mostly about the experimental evaluation.  1) I would have liked to see an experiment with categorical latent variables on a more "classical" model, similar to the ones considered in [9,32,6].  2) I was missing a comparison with other gradient estimators for non-reparameterizable distributions, such as REBAR [32] and RELAX [6]. Can you say something about how these methods would compare?  3) I was confused about the numbers in Table 2. What does it mean exactly "to complete a single optimization trajectory"? I suggest to replace these numbers with the time per iteration of each approach, because otherwise it is hard to read if the numbers presented involve different number of iterations in each cell of the table.  4) I appreciate the honesty about the computation time being the worst for the proposed approach. However I would have liked to see a better implementation rather than an early unoptimized prototype. Based on the behavior of the "Repar" method in Figure 1, my guess is that the suggested idea of subsampling the L terms of the boundary component will perform well enough. I also suggest to implement at least one simple model in a library other than Python+autograd that avoids dynamic computational graphs. Even if the resulting implementation is not generic, it will allow to learn something about the true computational cost of each approach.