From: ICLR.cc/2018/Conference/Paper958/Authors
title: Paper revision 2
comment: In addition to our revision 1, in which we extensively revised all experiments involving ZINC dataset, we have made an updated revision 2 which mostly addresses the writing and presentation issues. Besides the refinement of wording and typos, this version includes the following modification:

1) We added Figure 2, where we explicitly show how the modern compiler works through the example of two-stage check (i.e., CFG parsing and Attribute Grammar check). Section 2 is now augmented with more detailed explanations of background knowledge.

2) We added Figure 3, which shows the proposed syntax-directed decoder step by step through an example. Through the examples we put more effort in explaining key concepts in our method, such as ‘inherited constraints’ and ‘lazy linking’. 

3) Experiment section is revised with more details included. 

4) We added a conclusion section as suggested by the reviewer.

From: ICLR.cc/2018/Conference/Paper958/Authors
title: Reply to "Review"
comment: Thanks for your effort in providing this detailed and useful review! 

We present our clarification in the following:

>> Use of data and comparison with baselines:

We would first note that the anonymous accusation was set to “17 Nov 2017 (modified: 28 Nov 2017), readers: ICLR 2018 Conference Reviewers and Higher”. That’s why it was not visible to us until Nov 28, i.e., the original review release date. This gives us no chance to clarify anything before the review deadline. We have replied to it actively since Nov 28. 
**Note the thread is invisible to us again since Dec 2. **

1) We have experimented both kekulization and non-kekulization for baselines, and have reported the best they can get in all experiments. For example, in Table 2 the GVAE baseline results are improved compared to what was reported in GVAE paper.

2) The anonymous commenter is using different kekulization (RDKIT, rather than our used Marvin), different baseline implementation (custom implementation, rather than the public one in GVAE’s paper) and possibly different evaluation code (since there is no corresponding evaluation online). For a reproducible comparision, we released our implementation, data, pretrained model and evaluation code at:  https://github.com/anonymous-author-80ee48b2f87/cvae-baseline

3) To make further clarification, we ran our method on the vanilla (non-kekulised) data. Our performance is actually boosted (76.2% vs 72.8% reported in the paper).
The details of results from these experiments above can be seen in our public reply titled “We released baseline CVAE code, data and evaluation code for clarification” and “Our reconstruction performance without kekulization on Zinc dataset”. 

In either setting still, our method outperforms all baselines on reconstruction. We are sorry that this may have led to some confusions. To avoid further possible misunderstandings, we have extensively rerun all experiments involving ZINC dataset. Though differences are observed, the conclusion in each experiment remains the same. For example, our reconstruction performance is boosted (76.2% vs 72.8%). Since we didn’t address aromaticity semantics by the paper submission deadline, the valid prior fraction drops to 43.5%, but it is still much higher than baselines (7.2% GVAE, 0.7% CVAE). Please find the updated paper for more details. 

>> prior knowledge and limitations  

We are targeting on domains where strict syntax and semantics are required. For example, the syntax and semantics are needed to compile a program, or to parse a molecule structure. So such prior knowledge comes naturally with the application. Our contribution is to incorporate such existing syntax and semantics in those compilers, into an on-the-fly generation process of structures. 

In general, when numerous amount of data is available, a general seq2seq model would be enough. However, obtaining the useful drug molecules is expensive, and thus data is quite limited. Using knowledges like syntax (e.g., in GVAE paper), or semantics (like in our paper) will greatly reduce the amount of data needed to obtain a good model.

In our paper, we only addressed 2-3 semantic constraints, where the improvement is significant. Similarly, in “Harnessing Deep Neural Networks with Logic Rules (Hu et.al, ACL 16)”, incorporating several intuitive rules can greatly improve the performance of sentiment analysis, NER, etc. So we believe that, incorporating the knowledge with powerful deep learning achieves a good trade-off between human efforts and model performance. 

>> Typos and other writing issue:

We thank you very much for your careful reading and pointing out the typos and writing issues in our manuscript! We have incorporated your suggested changes in the current revision, and are keeping conducting further detailed proofreading to fix as much as possible the writing issues in the future revisions.

From: ICLR.cc/2018/Conference/Paper958/AnonReviewer2
title: Presentation improved but still lacking
comment: The presentation of the paper has definately improved, but I find the language used in the paper still below the quality needed for publication. There are still way too many grammatical and syntactical errors.

From: ICLR.cc/2018/Conference/Paper958/Authors
title: Fighting for our honestness
comment: We will not accept the accusation of “deliberately misleading and dishonest”. Please read the paper carefully before writing down the public comments of groundless moral judgement, which will lead to misunderstanding to public. 

First of all, we need to mention that, for the two baselines (CVAE and GVAE), **we also tried the kekule form of the data** along with **many other possible settings**, as we mentioned in the paper in page 15, right above Appendix C. We report whatever best result these baselines can achieve in our paper (page 8) from these settings. That’s why for GVAE, the result reported by us in Table 2 is **even better** than what reported in GVAE paper! Before going further, we would like to emphasize that these are our efforts to be honest with baselines.

Secondly, we are not reporting “valid reconstruction” as you mentioned. We are reporting two completely separate metrics namely (1) **exact** reconstruction and (2) valid **prior**, faithfully following the protocol in GVAE’s paper (in its Appendix C). From the number you are mentioned, we highly doubt that you are referring to accuracy in single character-level (not reconstructing entire SMILES), since we also get ~75% character accuracy for CVAE with kekule data. 

Thirdly, we don’t think the kekule form makes the problem simpler. (1) Take the benzene ring for example. Non-kekule form is “c1ccccc1” but the kekule form is “C1=CC=CC=C1”. The generator should output the alternating single and double bonds in the exact same way. This actually makes the problem harder for some baselines! (2) Also the SMILES space you mentioned “26^85 with kekulisation vs 36^120 without” is puzzling without further clarifying how it is calculated, as the magic numbers differ from statistics of the ZINC data, where in kekule data, the maximum length is similar (114 vs 120), while average length is longer but still close (48 vs 44). We used “Marvin (https://chemaxon.com/products)” to get the kekule form. (3) We don’t think kekulisation is “lossy” as you mentioned, at least for Zinc dataset. What we did is to get the canonical form (using RDKIT) of both kekule and normal SMILES, and we verified that for all SMILES the canonical form matched. We would argue that at least for our experiment that covers our method and previous baselines, the two forms have the same theoretical representation power in this dataset.

Given the above reasons, it is reasonable that the result we got for CVAE is much worse with kekule form under our metrics. Again, we emphasize the metric used in the code in CVAE is totally different. If you also use the CVAE code from https://github.com/mkusner/grammarVAE, then you will see it outputs something like [“loss”, “acc”]. We run the same code for kekule data, and it outputs “acc” roughly 75%, which is similar to what you mentioned. **However**, this is the single character-level accuracy, not the accuracy of recovering entire SMILES. Accuracy for successfully reconstructing the entire SMILES as a whole is almost zero as we tried (e.g., 0.75^100).

So we suggest to please make several things clear first for a constructive discussion:
1) Is the same Kekule form used? Our data has 114 max length but you reported 85.
2) Is the same CVAE code from grammarVAE used? 
3) Are we reporting the same metric? Since our single character-level accuracy matches your number which is roughly 75%, this may indicate a confusion in choice of accuracy. **Note that reconstructing the entire sequence as a whole exactly is significantly harder for sequence model**.

We reiterate that what you mentioned under the clarification above is already addressed in our paper. If you think your experiments are consistent with above (code, data, metric, etc), could you please kindly share your code and kekule data through anonymous link, so that we can calibrate our possible disagreement and make clear the possible issue? 

Finally we expect the anonymity can post the comments to public for timely discussion, otherwise, we cannot receive the update email from the OpenReview system in time, which may delay our reply and hurt the constructiveness of the discussion.

From: ICLR.cc/2018/Conference/Paper958/Authors
title: Our reconstruction performance without kekulization on Zinc dataset
comment: To further clarify the reconstruction accuracy, we here report performance (our model and baselines) without using the kekulization transformation on Zinc dataset, in supplement to numbers using kekulization already reported in our manuscript. We include baseline results from GVAE paper for direct comparison. 

SD-VAE (ours): 76.2%; GVAE: 53.7%; CVAE: 44.6%

Compare to what reported for SD-VAE with kekulization in current revision (72.8%), our performance is slightly boosted without kekulization. This shows that kekulization itself doesn’t have positive impact for reconstruction in our method. Our conclusion that the reconstruction accuracy of our SD-VAE is much better than all baselines still holds. 

Nevertheless, to avoid possible misunderstanding, we’ll refine the experiment section by including more experiments, once the open review system allows.

From: ICLR.cc/2018/Conference/Paper958/AnonReviewer2
title: Interesting idea but poor presentation
rating: 3: Clear rejection
review: The paper presents an approach for improving variational autoencoders for structured data that provide an output that is both syntactically valid and semantically reasonable.  The idea presented seems to have merit , however, I found the presentation lacking. Many sentences are poorly written making the paper hard to read, especially when not familiar with the presented methods. The experimental section could be organized better. I didn't like that two types of experiment are now presented in parallel. Finally, the paper stops abruptly without any final discussion and/or conclusion. 
confidence: 2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper

From: ICLR.cc/2018/Conference/Paper958/Authors
title: Paper revision 1
comment: To avoid further possible misunderstandings we have update our paper, in which we have extensively revised all experiments involving ZINC dataset. This addresses concerns on use of ZINC data and comparison with previous methods. 

The conclusion in each experiment **remains the same** though some differences are observed. Examples of differences are as following: Our reconstruction performance is boosted (76.2% vs 72.8%); And since we didn’t address semantics specific to aromaticity by the paper submission deadline, the valid prior fraction drops to 43.5%, but it is still much higher than baselines (7.2% GVAE, 0.7% CVAE).

Please find the updated paper for more details.

From: ICLR.cc/2018/Conference/Paper958/Authors
title: We released baseline CVAE code, data and evaluation code for clarification
comment: To address the anonymous commenter’s concerns on the CVAE baseline, the initial release of CVAE’s code (training code based on GVAE’s authors’code), with two versions of kekule data and vanilla data and the reconstruction evaluation script,  are available at 

https://github.com/anonymous-author-80ee48b2f87/cvae-baseline 

where we also uploaded our trained CVAE, together with pretrained model obtained from GVAE’s authors. 

Here we briefly summarize the current results:
(1) - CVAE, vanilla setting, pretrained model : 44.854%
(2) - CVAE, vanilla setting, our retraining: 43.218%
(3) - CVAE, Marvin Suite kekulised **tried for all methods in our paper**: 11.6%
(4) - CVAE, rdkit kekulised (provided by anonymous commenter, never been tried in our paper): 38.17% 

We reported the best form of SMILES for CVAE in our paper. If you believe there’s any issue, please let us know asap and we are happy to investigate.

Finally, we thank all the anonymous comments about the paper. If you have any concerns about the paper, please make the comments public while you specifying readers. Making such comments to reviewers only will not allow us to address the possible misunderstandings, or improve the paper timely when we make possible mistakes.

From: ICLR.cc/2018/Conference/Paper958/Authors
title: Discrepancy in data and evaluation code on where your concern is -- part 1
comment: We need to reiterate that for CVAE and GVAE, **we also tried the kekule form of the data** along with **many other possible settings**. In Table 1, GVAE got similar results as before, but CVAE got worse results. So we report whatever best result for the baselines, using the code from GVAE implementation. This is supported by the improvement of GVAE in Table 2. 

We never try to hide any details. Instead we elaborate all details and only due to the space limit, we put some of the stuffs in appendix (but still referred in main paper). That’s why you know details about datasets, settings, etc. So we suggest to figure out the discrepancy in scientific facts first, before making moral judgement. 

Please note that we mostly take issue with your argument is on the sequence reconstruct accuracy you get for baseline CVAE (75%) with your data, on top of which as a premise you argue that the performance we report may not surpass the baseline. However, its validity looks questionable to us, and we investigated into the details about your evidence by running the experiments on every possible settings including the one you use, and our finding **does not** support the soundness of your premise. We provided the codes and models that support our claim, and would like to calibrate this discrepancy first to make further discussion meaningful.

== Kekulisation (data):

We first introduce a basic common knowledge in Biochemistry that ``kekulisation form’’ is *NOT* unique. For the same underlying molecule, there may be many variants of kekulisation form due to different rules in rewriting SMILES, and it is pretty normal that two toolkits shows different kekulisations. 

We appreciate your providing the scripts. The script looks good, but it generates different data than ours. Your script uses kekulisation from **rdkit**, which leads to data form different from our kekulisation using **Marvin Suite**. This leads to the following discrepancy:
```
Non kekulised:
    max 120 min 9 mean 44.31076 std 9.32734
our kekulised:
    max 114 min 10 mean 48.73391 std 10.03919
your kekulised (as your script does):
    max 85 min 9 mean 44.85288 std 9.61040
```
This indicates even just from these numbers, non-kekulised and kekulised (both ours and yours) form of the ZINC are quite similar, in contrast to your argument that the later is much simpler. As we mentioned, alternating single and double bonds makes the kekule form much harder for some baselines. 

We would like to emphasize two points: First, both non-kekulised & kekulised are the representation corresponding to the same underlying molecules. Second, not only limited to our method but also for a large family of sequence models in applications as translation, summarization, etc., the distribution of length of sequence matters more than maximum length in analysis of data space, so we do not agree with your estimation of difficulty with a lsose bound inferred solely from maximum length.

From: ICLR.cc/2018/Conference/Paper958/AnonReviewer1
title: Review
rating: 5: Marginally below acceptance threshold
review: Let me first note that I am not very familiar with the literature on program generation, 
molecule design or compiler theory, which this paper draws heavily from, so my review is an educated guess. 

This paper proposes to include additional constraints into a VAE which generates discrete sequences, 
namely constraints enforcing both semantic and syntactic validity. 
This is an extension to the Grammar VAE of Kusner et. al, which includes syntactic constraints but not semantic ones.
These semantic constraints are formalized in the form of an attribute grammar, which is provided in addition to the context-free grammar.
The authors evaluate their methods on two tasks, program generation and molecule generation. 

Their method makes use of additional prior knowledge of semantics, which seems task-specific and limits the generality of their model. 
They report that their method outperforms the Character VAE (CVAE) and Grammar VAE (GVAE) of Kusner et. al.  
However, it isn't clear whether the comparison is appropriate: the authors report in the appendix that they use the kekulised version of the Zinc dataset of Kusner et. al, whereas Kusner et. al do not make any mention of this. 
The baselines they compare against for CVAE and GVAE in Table 1 are taken directly from Kusner et. al though. 
Can the authors clarify whether the different methods they compare in Table 1 are all run on the same dataset format?

Typos:
- Page 5: "while in sampling procedure" -> "while in the sampling procedure"
- Page 6: "a deep convolution neural networks" -> "a deep convolutional neural network"
- Page 6: "KL-divergence that proposed in" -> "KL-divergence that was proposed in" 
- Page 6: "since in training time" -> "since at training time"
- Page 6: "can effectively computed" -> "can effectively be computed"
- Page 7: "reset for training" -> "rest for training" 
confidence: 1: The reviewer's evaluation is an educated guess

From: ICLR.cc/2018/Conference/Paper958/Authors
title: Reply to "Interesting idea but poor presentation"
comment: We thank you for providing reviews.

We’ll refine the paper to include more introduction about background, and more detailed explanations about our method. 

We’ll include final discussion/conclusion section.

From: ICLR.cc/2018/Conference/Paper958/AnonReviewer3
title: Strong paper presents state-of-the-art results
rating: 7: Good paper, accept
review: NOTE: 

Would the authors kindly respond to the comment below regarding Kekulisation of the Zinc dataset? Fair comparison of the data is a serious concern. I have listed this review as a good for publication due to the novelty of ideas presented, but the accusation of misrepresentation below is a serious one and I would like to know the author's response.

*Overview*

This paper presents a method of generating both syntactically and semantically valid data from a variational autoencoder model using ideas inspired by compiler semantic checking. Instead of verifying the semantic correctness offline of a particular discrete structure, the authors propose “stochastic lazy attributes”, which amounts to loading semantic constraints into a CFG and using a tailored latent-space decoder algorithm that guarantees both syntactic semantic valid. Using Bayesian Optimization, search over this space can yield decodings with targeted properties.

Many of the ideas presented are novel. The results presented are state-of-the art. As noted in the paper, the generation of syntactically and semantically valid data is still an open problem. This paper presents an interesting and valuable solution, and as such constitutes a large advance in this nascent area of machine learning.

*Remarks on methodology*

By initializing a decoding by “guessing” a value, the decoder will focus on high-probability starting regions of the space of possible structures. It is not clear to me immediately how this will affect the output distribution. Since this process on average begins at high-probability region and makes further decoding decisions from that starting point, the output distribution may be biased since it is the output of cuts through high-probability regions of the possible outputs space. Does this sacrifice exploration for exploitation in some quantifiable way? Some exploration of this issue or commentary would be valuable. 

*Nitpicks*

I found the notion of stochastic predetermination somewhat opaque, and section 3 in general introduces much terminology, like lazy linking, that was new to me coming from a machine learning background. In my opinion, this section could benefit from a little more expansion and conceptual definition.

The first 3 sections of the paper are very clearly written, but the remainder has many typos and grammatical errors (often word omission). The draft could use a few more passes before publication.

confidence: 3: The reviewer is fairly confident that the evaluation is correct

From: ICLR.cc/2018/Conference/Program_Chairs
title: ICLR 2018 Conference Acceptance Decision
comment: This paper presents a more complex version of the grammar-VAE, which can be used to generate structured discrete objects for which a grammar is known, by adding a second 'attribute grammar', inspired by Knuth.

Overall, the idea is a bit incremental, but the space is wide open and I think that structured encoder/decoders is an important direction.  The experiments seem to have been done carefully (with some help from the reviewers) and the results are convincing.
decision: Accept (Poster)

From: ICLR.cc/2018/Conference/Paper958/Authors
title: Reply to "Strong paper presents state-of-the-art results"
comment: Thanks for your effort in providing this detailed and constructive review! 
We present our clarification in the following:

>>NOTE:

We would first note that the anonymous accusation was set to “17 Nov 2017 (modified: 28 Nov 2017), readers: ICLR 2018 Conference Reviewers and Higher”. That’s why it was not visible to us until Nov 28, i.e., the original review release date. This gives us no chance to clarify anything before the review deadline. We have replied to it actively since Nov 28. 
**Note the thread is invisible to us again since Dec 2. **

To summarize our clarification: 

>> Use of data

1) We have experimented both kekulization and non-kekulization for baselines, and have reported the best they can get in all experiments. For example, in Table 2 the GVAE baseline results are improved compared to what was reported in GVAE paper.

2) The anonymous commenter is using different kekulization (RDKIT, rather than our used Marvin), different baseline implementation (custom implementation, rather than the public one in GVAE’s paper) and possibly different evaluation code (since there is no corresponding evaluation online). For a reproducible comparision, we released our implementation, data, pretrained model and evaluation code at:  https://github.com/anonymous-author-80ee48b2f87/cvae-baseline

3) To make further clarification, we ran our method on the vanilla (non-kekulised) data. Our performance is actually boosted (76.2% vs 72.8% reported in the paper).
The details of results from these experiments above can be seen in our public reply titled “We released baseline CVAE code, data and evaluation code for clarification” and “Our reconstruction performance without kekulization on Zinc dataset”. 

In either setting still, our method outperforms all baselines on reconstruction. We are sorry that this may have led to some confusions. To avoid further possible misunderstandings, we have extensively rerun all experiments involving ZINC dataset. Though differences are observed, the conclusion in each experiment remains the same. For example, our reconstruction performance is boosted (76.2% vs 72.8%). Since we didn’t address aromaticity semantics by the paper submission deadline, the valid prior fraction drops to 43.5%, but it is still much higher than baselines (7.2% GVAE, 0.7% CVAE). Please find the updated paper for more details. 

>>sacrifice of exploration

CVAE, GVAE and our SD-VAE are all factorizing the joint probability of entire program / SMILES text in some way. CVAE factorizes in char level, GVAE in Context Free Grammar (CFG) tree, while ours factorizes both CFG and non-context free semantics. Since every method is factorizing the entire space, each structure in this space should have the possibility (despite its magnitude) of being sampled. 

Bias is not always a bad thing. Some bias will help the model quickly concentrate to the correct mode. Definitely, different methods will bias the distribution in a different way. For example, CVAE is biased towards the beginning of the sequence. GVAE is biased by several initial non-terminals.  

Our experiments on diversity of generated molecules (table 3) demonstrate that, both GVAE and our method can generate quite diverse molecules. So we think both methods don’t have noticeable mode collapse problem on this dataset.

>> writings:

Thanks for the suggestions. We are adding more effort in explaining our algorithm and improve writing in revisions. We have revised our experiments sections for clarifying the most important issue, and will keep improving the writing.

To briefly answer the “lazy linking”: We don’t sample the actual value of the attribute at the first encounter; Instead, later when the actual content is generated, we use bottom-up calculation to fill the value. For example, when generating ringbond attribute, we only sample its existence. The ringbond information (bond index and bond type) are filled later. 

As a side note, this idea comes from “lazy evaluation” in compiler theory where a value is not calculated until it is needed.

From: ICLR.cc/2018/Conference/Paper958/Authors
title: Discrepancy in data and evaluation code on where your concern is -- part 2
comment: (Due to the space limit, see comment below for part 1)

== Code:

As mentioned before, for training CVAE baseline we use the code from GVAE (https://github.com/mkusner/grammarVAE) instead of making an implementation in other framework. **However, the reconstruction evaluation code is not available in this repo. Thus we made our evaluation code following GVAE paper’s protocol , and made it public.**

What we achieved with CVAE and our evaluation code for exact sequence reconstruction is:
```
(1) - CVAE, vanilla data, pretrained model : 44.854% (As we showed in the paper)
(2) - CVAE, vanilla data, our retraining: 43.218%
(3) - CVAE, our kekulised data: 11.6%
(4) - CVAE, your kekulised data: 38.17% 
```
With the following scripts (If you want to evaluate on your own, please put them on the GVAE’s repo and run the first one after modifying paths to model weight / data files):
https://gist.github.com/anonymous-author-80ee48b2f87/9f025de58123450c15e8d95d29797826
https://gist.github.com/anonymous-author-80ee48b2f87/aa0ff838eabc372d537c57199ebc31f4

Note that (1) is consistent with what GVAE paper reports, and (2) is consistent with (1), which gives us confidence on the correctness of both our evaluation script and our running of code from GVAE’s repo.

See NOTE 1 below for everything needed to reproduce our evaluation.

FOCUS:

The observation is that CVAE performs worse on kekulised data (  (1, 2)  vs. (3, 4) in evaluation above), as we have mentioned before, **in contrast to** receiving huge performance gain to become a strong baseline ( (1, 2) vs. 75% as you have claimed).  Again, in our paper, we ran the baselines on both settings and report the best results!

So new we observe discrepancy in the data and the metric evaluation between our findings and your argument. Since data and metric are keystones for your argument, we would like to verify it by asking you how do you **in detail** get your number (75%). Could you please point to the code for evaluation, CVAE implementation for training and data you used for a fair comparison? 

NOTE 1: the initial release of baseline’s code (with two versions of kekule data and vanilla data and the reconstruction accuracy evaluation script; training code based on GVAE’s authors’code)  are available at https://github.com/anonymous-author-80ee48b2f87/cvae-baseline , where we also uploaded our trained CVAE, together with pretrained model obtained from GVAE’s authors. 

NOTE 2: Before getting numbers for the experiment above, we fixed a problem in our script that collects statistics for the previous reply, so now the reported sequence level accuracy for CVAE on kekulised data is ~11.6%, and character level accuracy is >97%. Since this is a bug in collecting numbers rather than the model, our conclusion, which says kekulisation makes it harder for CVAE instead of making it perform better as you argued, remains **the same**.

== Final words:

Finally, thank you for the follow up. But more details about what you’ve tried are required, which is necessary for the scientific discussion among us. We look forward to calibrating mismatching of numbers in the metric which are fundamentals of further discussion.

Nevertheless, we will also report the results of our method on the vanilla settings in the revision of our manuscript, once allowed by openreview.