Paper:
Generative models have achieved impressive results in many domains including image and text generation. In the natural sciences, generative models have lead to rapid progress in automated drug discovery. Many of the current methods focus on either 1-D or 2-D representations of typically small, drug-like molecules. However, many molecules require 3-D descriptors and exceed the chemical complexity of commonly used datasets. We present a method to encode and decode the position of atoms in 3-D molecules along with a dataset of nearly 50,000 stable crystal unit cells that vary from containing 1 to over 100 atoms. We construct a smooth and continuous 3-D density representation of each crystal based on the positions of different atoms. Two different neural networks were trained on a dataset of over 120,000 three-dimensional samples of single and repeating crystal structures. The first, an Encoder-Decoder pair, constructs a compressed latent space representation of each molecule and then decodes this description into an accurate reconstruction of the input. The second network segments the resulting output into atoms and assigns each atom an atomic number. By generating compressed, continuous latent spaces representations of molecules we are able to decode random samples, interpolate between two molecules, and alter known molecules.

Review:
PAPER SUMMARY: This paper addresses the problem of encoding and decoding 3D chemical structures, with the ultimate goal of generating 3D crystal structures. The authors propose an auto-encoder framework for encoding the 3D locations of atoms in the crystal to a latent representation and then decoding that representation back into 3D structure. The paper's contributions can be summarized as follows:
1) A data representation that converts the 3D atom locations to a 3D voxel density map, so that they can be encoded by a standard 3D convolutional network.
2) A decoder network that first estimates a 3D density map from the latent vector using upsampling and convolutions and then classifies the atom type (atomic number) per voxel using a 3D segmentation.
3) The network is applied on unit cells of crystals or repeated unit cells from a dataset of crystal structures and the network is shown to be able to accurately reconstruct the 3D density maps, predict the number of atoms in the cell and perform fairly well in their classification into atomic types.
I appreciate that the paper addresses an interesting problem that is not sufficiently explored and can motivate the development of novel methods that generate 3D molecules with particular structure and multiple types of atoms, however the current work combines existing methods, without any architectural modifications that exploit the new domain.  If the presentation of results and experiments is improved, this could be a good application paper in material science with an interesting combination of techniques from machine learning and computer vision. It would be more appropriate to submit this paper to a domain-specific venue rather than to ICLR. Therefore, I cannot justify its acceptance to ICLR in its current format.
Strengths
-----------------------
1.	New domain: The paper addresses an interesting problem in a new domain. Previous work on generative models for molecules have concentrated on molecules with 1D structure (which can be represented as strings) or 2D structure (which can be represented as planar graphs). Generating 3D molecule structures is a challenging and interesting problem.
2.	Efficient data representation: The authors bypass the difficulties associated with modeling sets of 3D points with arbitrary structure by proposing a canonical, voxel-based density representation.
3.	Joint VAE + UNet training: The joint training of the encoder-decoder VAE and the 3D segmentation network results in a decoder that can reconstruct atom locations and atom types, being robust to mistakes in the density map reconstruction.
4.	Qualitative results: There are nice visualizations of the reconstructed molecules and of the effect of the latent variable z. Figure 3 in particular does a great job at elucidating the outputs of the network and the reconstruction quality.
Weaknesses
------------------------
1.	Limited methodological novelty: The methods used in the paper, i.e. the data representation (see detailed comments), the encoder network, the decoder network and the segmentation network are all existing methods without any (or with only minor) modifications.  Their combination also seems straightforward, except for the joint training of the VAE and the segmentation network, which has not been tried before to the best of my knowledge. Networks have not been modified to exploit the intricacies of the new domain/task, such as symmetries, repetition of unit cells, large number of atoms.
2.	Lack of comparison with other methods/baselines: There is no quantitative comparison with alternative baselines or methods or even discussion of such alternatives. It is understandable that the paper addresses a relatively new problem, however the method could be compared to CrystalGAN. Also, there is no justification of the advantages of using a voxel-based density map representation along with 3D convolutions vs, for instance, a graph representation along with graph convolutions (Xie & Grossman, [3]) or a point cloud GAN (Achlioptas). 
3.	No quantitative evaluation of the generative capabilities of the network: In the case of drug molecule generation, logP and QED scores  [4,5] are used to evaluate their drug-likeness. Without such scores in the crystal generation domain, it is not easy to judge how good a generated crystal is.  The need for some type of quantitative evaluation is especially important, since it is not as easy to qualitatively judge the quality of crystals obtained by sampling random latent vectors (Fig 5B) as in the case of generated images or text. The only such evaluation presented in the paper is related to the distribution of distances between atoms which seems to be close to their actual distribution in nature. 
4.	Presentation of quantitative results: The quantitative results are scattered in text throughout the results section without being summarized in a table. Results about the distance of generated atoms w.r.t to their true location, the predicted atom counts, predicted atom types etc. for the cases of single unit cells and repeated unit cells should be added to a table to complete a rigorous experimental evaluation. 
Additional Comments
-------------------------
1.	Lack of novelty: The data representation, which is presented as one of the core contributions of the paper, is not new. Radial basis functions have been used to generate 3D density maps (e.g. [2]) and 3D Convolutional VAEs (Brock2016, [1]) have been employed on density maps (e.g. occupancy maps) on computer vision tasks. 
2.	Confusing description of the repeated unit cells case: It is not clear, especially for a general audience, how the repeated unit cells data are generated (in which directions are the unit cells repeated, how many times). It is also not clear how the training routine (conditioning) and the output post-processing (connected components + majority voting) is modified.
3.	After Eq. (2), q is the --approximate-- posterior …
Suggested References
--------------------------
1.	VoxNet: A 3D Convolutional Neural Network for Real-Time Object Recognition
2.	VV-NET: Voxel VAE Net with Group Convolutions for Point Cloud Segmentation
3.	Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs
4.	Graph Convolutional Policy Network forGoal-Directed Molecular Graph Generation
5.	Grammar Variational Autoencoder
6.	3D U-Net: Learning Dense VolumetricSegmentation from Sparse Annotation
7.	PointConv: Deep Convolutional Networks on 3D Point Clouds