Paper:
For reliable transmission across a noisy communication channel, classical results from information theory show that it is asymptotically optimal to separate out the source and channel coding processes. However, this decomposition can fall short in the finite bit-length regime, as it requires non-trivial tuning of hand-crafted codes and assumes infinite computational power for decoding. In this work, we propose Neural Error Correcting and Source Trimming (NECST) codes to jointly learn the encoding and decoding processes in an end-to-end fashion. By adding noise into the latent codes to simulate the channel during training, we learn to both compress and error-correct given a fixed bit-length and computational budget. We obtain codes that are not only competitive against several capacity-approaching channel codes, but also learn useful robust representations of the data for downstream tasks such as classification. Finally, we learn an extremely fast neural decoder, yielding almost an order of magnitude in speedup compared to standard decoding methods based on iterative belief propagation.

Review:
Summary of paper: For the finite-bit case of the noisy communication channel model, it is suboptimal to optimize source coding (compression of input) and error correction (fault tolerance for inherent noise in the channel) separately. The authors propose a neural network model (NECST) that is very similar to the standard VAE, except using binary latents with corruption (e.g.,  random bit flipping in the style of a binary symmetric channel). They use VIMCO to optimize through the discrete units. In their experiments, they show that they can outperform a JPEG+ideal channel code model, but perform similarly to a VAE+LDPC (LDPC is a classic error correcting code) setup.
First of all, the paper is quite well written and easily readable. Great work on explaining the motivation and the model -- the writing is clear and explains background knowledge extremely well.
The main contribution in the model is the use of discrete binary latents, instead of the standard continuous latents in a VAE. However, I am uncertain about the novelty of this contribution. There have been numerous works examining discrete latent variables in autoencoders (a random sampling: [1, 2, 3, 4]) and beyond. Furthermore, the method of training through discrete latents is also standard (VIMCO, though one can also imagine using more recent advances like REBAR or RELAX). The only difference would be the addition of noise to the discrete. I would be curious to see how that compares to recent works that have also added noise to discrete latents [5].
Thus, it strikes me that the main contribution of this work would be in comparing against the current best techniques for coding. However, the experiments section is weak, and does not provide significant evidence that the NECST model is better than the alternatives. NECST outperforms JPEG+ideal channel coding, but doesn't do much better than a VAE+LDPC baseline. This suggests that most of the gains comes from the encoder (source coding) model q(\hat{y} | x), instead of the joint training of source coding and error correcting code. It is not surprising that using a neural network to generate codes would provide significant gains. It's not clear that error correcting code aspect (noise in the latents) is particularly important.
Furthermore, in the classification results, the MLP model trained on the discrete codes gets 93% accuracy on noiseless MNIST inputs. You can easily get this accuracy by training logistic regression directly on the pixels. Despite what the authors write, this result suggests that the codes are not very useful for downstream learning. Furthermore, it is unclear why adding random noise to the inputs would significantly improve some of the weaker classifiers. The only reason I can think of is data augmentation, but this has nothing to do with the NECST model.
In conclusion, this is a well written paper, but the novelty is not apparent and the experimental results are weak, and so I am not convinced this is suitable for ICLR.
Additional Questions:
* How is the runtime computed? Specifically, for NECST, do you batch the data and then divide the forward pass time by the batch size? If this is how runtime is computed, it's not surprising that NECST does better, given that batching is cheap with modern hardware. If the actual forward pass time for a single example is cheaper than that of LDPC's belief propagation, then that would be quite promising.
* The authors state that VAEs optimize a lower bound on the marginal log-likelihood p(X), whereas NECST optimizes a lower bound on the mutual information I(X, Y), where Y is the noised code. The authors however do not discuss why one should optimize for mutual information compared to marginal log-likelihood. What are the advantages and disadvantages between the two?
[1] Semi-Supervised Learning with Deep Generative Models (https://arxiv.org/abs/1406.5298)
[2] Discrete Variational Autoencoders (https://arxiv.org/abs/1609.02200) 
[3] Neural Discrete Representation Learning (https://arxiv.org/abs/1711.00937)
[4] Discrete Autoencoders for Sequence Models  (https://arxiv.org/abs/1801.09797)
[5] Theory and Experiments on Vector Quantized Autoencoders (https://arxiv.org/pdf/1805.11063.pdf)