Paper:
We introduce the notion of lattice representation learning, in which the representation for some object of interest (e.g. a sentence or an image) is a lattice point in an Euclidean space. Our main contribution is a result for replacing an objective function which employs lattice quantization with an objective function in which quantization is absent, thus allowing optimization techniques based on gradient descent to apply; we call the resulting algorithms dithered stochastic gradient descent algorithms as they are designed explicitly to allow for an optimization procedure where only local information is employed. We also argue that a technique commonly used in Variational Auto-Encoders (Gaussian priors and Gaussian approximate posteriors) is tightly connected with the idea of lattice representations, as the quantization error in good high dimensional lattices can be modeled as a Gaussian distribution. We provide experimental evidence of the potential of using lattice representations by implementing a VAE that employs a rectangular lattice and is trained using our results, and contrasting its performance with that of a Gaussian VAE.

Review:
This paper proposes a framework for gradient descent optimization of latent variable models where the latent code lies on a (n-dimensional) lattice. After a brief overview of relevant results in lattice theory, a method ("Dithered gradient descent") is proposed to differentiate through lattice quantization by means of an additive "dither" noise variable and a lower bound on the total reconstruction+representation loss using a prior distribution. After elaborating on the relationship between their framework and the VAE literature, the authors go on to give an explanation of the link between the covering property of a lattice and the reconstruction loss. Finally, empirical evidence is given for the better compression and reconstruction properties of a sentence autoencoder with lattice latent codes.
Overall, while the idea presented in this paper is interesting (using lattices as discrete latent variables), I don't think that the paper in its current state is up to standards for ICLR.
First, there is little motivation for using lattices over other discrete latent variable models in the first place (eg. straight-through estimator with Gumbel softmax or VQ-VAE).
Second, while I appreciate the efforts that the authors have gone through to make lattice theory accessible to a profane audience, I think that the presentation (particularly in 2.2) could benefit from significant re-ordering as well as more emphasis on the parallel with the VAE literature. A lot of the concepts introduced there have equivalents in VAE lingo (prior, ELBO) and would benefit from being identified as such as they are introduced (and not just in a following sub-section). I also think that presenting Theorem 1 first is confusing, because we are shown the solution before seeing the problem. I think that it would be much more natural to 1. recall the original objective 2. introduce (and motivate) tithering and the $\hat X$ and then 3. introduce theorem 3 to explain how to get rid of quantization. In terms of global structure the paper is also rather imbalanced with 3 large sections including a monolithic 3 page long introduction. I would advise splitting up as appropriate to give the reader some space to breathe.
Third, and perhaps most importantly, the experimental section is underwhelming. The description of the method used is too sparse and ridden with inaccuracies and vague descriptions. A reader would be hard-pressed to try to reproduce these results from the paper alone. The task itself is rather arbitrary (sentence autoencoding with GRUs), and from what I understand, there is no direct comparison with the relevant literature (eg. Gumbel straight-through or VQ-VAEs). I think that the authors should have at least illustrated the appeal of lattice representation learning on a toy example. Furthermore as stated by the author in the conclusion, the experiments don't even involve actual finite dimension lattices, making most of the paper seem irrelevant in hindsight. Overall this section feels rushed and it is the biggest point against the paper for me. 
Finally, the paper is littered with small typos or unfortunate notation collisions which, if individually benign, make parsing this already dense paper harder than needed. Some specific examples are given in the notes at the end of my review.
Notes (in no specific order):
- Second to last paragraph on page 3: "one may want to have $\Delta>0$ be very small [...] but that leads to a larger reconstruction cost". I assume you meant "representation cost".
- $- \log p$ instead of $\log \frac 1 {p}$ to save space
- Voronoi cells are introduced but never referenced.
- Subscripted lowercase f is overloaded, use lowercase p or q for densities. This makes some of the formulas in 2.2 particularly annoying to parse.
- In 2.4 $G_1$ refers to something different than $G_m$ with $m=1$
- The term "dither" is not defined explicitly. The closest I could find is a reference to "the dither value U.". Given the prominence of the term in the paper (and in the name of your proposed method) I think this warrants more attention.
- In the experimental section you justify the use of Gaussian dithering (instead of the uniform dithering used in all the theorems) by "The justification for using Gaussian dithers is partly contained in the high resolution analysis done in Subsection 2.4.". However there is no explicit reference to this fact in 2.4 (in fact the words "gaussian" or "normal" don't even appear in 2.4). The link should be made more explicit somewhere.
- Some sentences seemingly don't make sense: "The representation cost for Gaussian dithering using techniques from Variational Autoencoders; in essence we assume an isotropic unit variance Gaussian as a prior over the representation space and then estimate the KL divergence between the dithered representation conditional on the encoder output and the Gaussian prior." in the experimental section.
- Page 3 codelength: references (eg. the shannon paper). In general a lot of the "It is well-known" statements in the paper should be accompanied by a reference.
- Some references don't have a date (Van den Oord in page 9 for instance)
- the result figure is very hard to read especially on paper. A bit of color as well as larger font/linewidth would go a long way.