Paper: Form follows Function: Text-to-Text Conditional Graph Generation based on Functional Requirements (ICLR-2024)

Reviewer comment (Experiments):
BLOOM is not a suitable baseline for molecular generation since it is not trained on molecular data. Galactica (Taylor et al., 2022) would be better since its training data includes smile strings. The simple autoregressive model can outperform the proposed message passing by increasing its training set.

Author response:
[Authors] While the experiments for this paper focus on the application of molecule generation, the focus of the paper itself is not a new method for performing molecule generation. The focus is a new method for generating graphs through serialized graph generation that match conditional functional requirements. Why do you feel that BLOOM as a language model is not suitable for the general problem of generating graphs through serialized graph generation that match conditional functional requirements?

Reviewer comment (Writing):
The language in the paper needs to be further polished.

Author response:
Given this comment, are there are any specific areas of the paper you would like us to focus on? Or possibly a specific way we explain things that you think could be improved?

Reviewer comment (Novelty):
The technical contribution is also limited. The primary contribution emphasized by the authors involves the use of causal masks during generation. However, the employment of causal masks for autoregressive generation is just a trick to ensure efficient training. (Otherwise, it will require to train on each token independently with multiple passes.) It's weird to label this well-known technical detail as the key contribution.

Author response:
We politely disagree with the claim that causal masking is just a trick to ensure efficient training. In fact the time complexity of causal masking is the same as no masking. In addition, a model trained without causal masking will perform very poorly when generating text because without masking the model has access to the next token it is trying to predict during training, so it does not learn anything. Novel attention masking methods have been published in ICLR, ICML and NeurIPS before, so masking methods are considered a contribution. The masking method we are proposing allows the model to incorporate graph structure into the inference process which is a contribution because to the best of our knowledge it has not been proposed before.

Reviewer comment (Theory):
The paper does not explain the reasoning for formulating the training objective as it is in Eqn 1. This is surprising given that experiments in Appendix D show a difference when using the differing term, but no satisfactory reason was given for why this might be the case.

Author response:
Thank you for pointing out the lack of clarity in our explanation. Upon reflection, it would have been best to introduce Eqn 2 first (the standard autoregressive training objective), then explain its deficiency with respect to the structure of the data and then introduce Eqn 1 (how we are proposing to train the model) as a objective which does taken into the structure of the data. This could make it clearer the reason for formulating Eqn 1. Instead we introduce Eqn 1 and then Eqn 2 and explain the difference between them with respect to the structure of our data.  Do you think this approach would improve the clarity of the objective?

Reviewer comment (Experiments):
Baseline without fine-tuning: Under Appendix D, how was the SSG-LLM w/out fine-tuning prompted? Was there any investigation done using in-context learning / few-shot prompting to get a parsable generation?

Author response:
SGG-LLM w/out fine-tuning used the same prompts as with fine tuning, which requested a molecule with a specific functional property. Please note, models that are performant at in-context learning like GPT-4 or Llama-2 were too large to fine tune given cost and hardware constraints. However, qualitative experiments were performed to see if it was possible for GPT-4 to generate molecules with a requested number of valence electrons and it was only able to produce molecules which that consisted of a single atom. We will do our best to extend  the results to include performant in-context learning LLMs for the camera ready if given the opportunity.

Reviewer comment (Theory):
As per Figure 1, is the reason that graph masking includes a self node (node i attends to node i) the same reason that causal masking has a token attending to itself? Is there any specific reason not to exclude that self node?

Author response:
We think there is a conceptual reason which is not very strong and an experimental design reason which is strong. The conceptual reason is that if the self node is excluded, so is its representation from the new representation calculated by the attention layer. So some information about the node itself is excluded from the new representation. This argument is weakened by the fact that most message passing and attention layers contain skip connections, but still holds water. The experimental design reason is that all prior work includes the self node, so excluding it would be another element of the method we would have to justify. However the inclusion of a self node in the attention or message passing layers is not directly related to the problem this paper is addressing, so it would not strengthen the paper to include it.

Reviewer comment (Experiments):
The feature vector for a node vector was selected as the feature vector of the last element describing that node. Were there any experiments done to vary this, for e.g. mean-pooling?

Author response:
No experiments were performed for other aggregation operations, but we chose this method because state-of-the-art sentence embedding techniques use the same approach. We leave it as future work to determine if there is a more performant aggregation method.

Reviewer comment (Experiments):
Why was BLOOM selected as the LLM backbone?

Author response:
Prior work on serialized graph generation used the LLM T5, we found this model performed very poorly on the task of molecule generation. So we tried using BLOOM in experiments and it had good performance.

Reviewer comment (Evaluation):
With reference to Table 3, why was parsability affected when using message passing?

Author response:
Since sampling is a stochastic process and the difference in parsability is less than or equal 0.2% in all cases, it is hard to tell if message passing had any impact or if it was an artifact of the sampling process. All methods generated at least 998 /1000 parsable graphs. Even SGG-LLM w/o message passing did not have a parsability of 100% on one of the data sets, so a model without message passing also did not achieve 100% parsability.