Paper: Text2Data: Low-Resource Data Generation with Textual Control (ICLR-2024)

Reviewer comment (Experiments):
There is no comparison to other techniques like data augmentation or transfer learning for low-resource text-to-data generation. Comparisons would be useful to know if Text2Data outperforms them.

Author response:
* Data Augmentation: we employed GPT-4 to modify the textual descriptions to augment the text-data pairs (EDM-DA), and we directly performed classifier-free guidance diffusion on molecular generation using EDM. Below are the results that evaluate the controllability of EDM on properties of generated molecules (Alpha and HOMO). 

| %  | Alpha     |              |        |        | HOMO      |              |        |        |
|----|-----------|--------------|--------|--------|-----------|--------------|--------|--------|
|    | Text2Data | EDM-Finetune | EDM-DA | EDM    | Text2Data | EDM-Finetune | EDM-DA | EDM    |
| 2  | 0.6636    | 0.8378       | 0.8002 | 0.8339 | 0.5792    | 0.9168       | 0.9534 | 1.0049 |
| 4  | 0.6206    | 0.7446       | 0.6855 | 0.6874 | 0.5583    | 0.8384       | 0.9301 | 1.0708 |
| 6  | 0.4384    | 0.6783       | 0.6736 | 0.6917 | 0.4364    | 0.9226       | 0.9005 | 0.9996 |
| 8  | 0.5247    | 0.6687       | 0.6600 | 0.6868 | 0.4658    | 0.9008       | 0.8816 | 0.9477 |
| 10 | 0.3412    | 0.6588       | 0.6128 | 0.7473 | 0.3299    | 0.8915       | 0.8825 | 0.9156 |

Based on the results above, data augmentation-based approach slightly improves performance of directly finetuning the model. This is potentially due to the improvement of generalization of text descriptions. 

* We have already performed transfer learning based approach in our paper. For instance, EDM-Finetune represents the model that EDM is pretrained on a large amount of molecular structures, and then finetuned on a small amount of text-molecule pairs. Based on the results above, EDM-Finetune has much worse performance compared with the proposed approach. This is expected due to the potential catastrophic forgetting of transfer learning-based approach.

Reviewer comment (Experiments):
The threshold ξ for the constrained optimization is set heuristically based on the minimum value of unsupervised loss L1.

Author response:
* The threshold does not have to be set strictly based on the minimum value of unsupervised loss L1, but can be further relaxed as long as it lies in the confidence bound of the minimum value of unsupervised loss L1.

* To guide the relaxation of the threshold, we have derived a generalization bound as shown in Theorem 1. In Theorem 1, we have proved that the threshold can be further relaxed as long as it lies in the summation of empirical minimum value of unsupervised loss L1 and the confidence bound $\epsilon=\epsilon_N+\epsilon_{N_p}$. In practice, we multiply a hyperparameter before the threshold to relax it. We have highlighted relevant part in red in the main paper.

Reviewer comment (Experiments):
My mainly technical concern on this paper is mainly on how to come up with thresholds for constrained optimization.

Author response:
* The threshold does not have to be set strictly based on the minimum value of unsupervised loss L1, but can be further relaxed as long as it lies in the confidence bound of the minimum value of unsupervised loss L1.

* To guide the relaxation of the threshold, we have derived a generalization bound as shown in Theorem 1. In Theorem 1, we have proved that the threshold can be further relaxed as long as it lies in the summation of empirical minimum value of unsupervised loss L1 and the confidence bound $\epsilon=\epsilon_N+\epsilon_{N_p}$. In practice, we multiply a hyperparameter before the threshold to relax it. We have highlighted relevant part in red in the main paper.

Reviewer comment (Novelty):
As for the method, the proposed method is relevant to the classifier-free guidance, which both involves a conditional model and an unconditional model (and the two models share the parameters). The difference is that the proposed method: finetune the unconditional model while adding constraint for catastrophic forgetting and the classifier-free guidance: guidance the denoising direction (in diffusion model) of conditional model with the help of unconditional model. I think the discussion of why the proposed method is better than the classifier-free guidance is necessary.

Author response:
We target the situation while labeled data is very limited. Directly learning a conditional model on such limited data may not have satisfactory generalizability and suffer from the limited diversity of generated data. Additionally, as there are too many parameters in the diffusion model (e.g., 14 million for Motion Diffusion Model), the lack of labeled data may cause issues such as overfitting. By contrast, our model is first pre-trained on unlabeled data, which usually ensures enough sample size to avoid overfitting.

Reviewer comment (Experiments):
As for the related work, the paper should discuss the related work about transfer learning, where catastrophic forgetting is an important topic. Since one of the contributions of the paper is preventing catastrophic forgetting. I think the paper should compare the proposed method with other works about catastrophic forgetting.

Author response:
* We have discussed representative works of transfer learning and its potential drawbacks in causing catastrophic forgetting. We have highlighted this part in red for easier reference:

“Tu et al. (2019) proposes to learn a mapping between source and target linguistic symbols and employs transfer learning to transfer knowledge from a high-resource language to low-resource language.”

“Nevertheless, all those strategies have their own limitations, such as high computational complexity for data augmentation, difficulty in maintaining the correct interpretation of text when leveraging unlabeled data during semi-supervised learning, and the potential catastrophic forgetting issues in transfer learning.”

* We have also conducted additional experiments for data augmentation, another approach to overcome catastrophic forgetting.

Specifically, we employed GPT-4 to modify the textual descriptions to augment the text-data pairs (EDM-DA), and we directly performed classifier-free guidance diffusion on molecular generation using EDM. Below are the results that evaluate the controllability of EDM on properties of generated molecules (Alpha and HOMO). 

| %  | Alpha     |              |        |        | HOMO      |              |        |        |
|----|-----------|--------------|--------|--------|-----------|--------------|--------|--------|
|    | Text2Data | EDM-Finetune | EDM-DA | EDM    | Text2Data | EDM-Finetune | EDM-DA | EDM    |
| 2  | 0.6636    | 0.8378       | 0.8002 | 0.8339 | 0.5792    | 0.9168       | 0.9534 | 1.0049 |
| 4  | 0.6206    | 0.7446       | 0.6855 | 0.6874 | 0.5583    | 0.8384       | 0.9301 | 1.0708 |
| 6  | 0.4384    | 0.6783       | 0.6736 | 0.6917 | 0.4364    | 0.9226       | 0.9005 | 0.9996 |
| 8  | 0.5247    | 0.6687       | 0.6600 | 0.6868 | 0.4658    | 0.9008       | 0.8816 | 0.9477 |
| 10 | 0.3412    | 0.6588       | 0.6128 | 0.7473 | 0.3299    | 0.8915       | 0.8825 | 0.9156 |

Based on the results above, data augmentation-based approach slightly improves performance of directly finetuning the model. This is potentially due to the improvement of generalization of text descriptions.

Reviewer comment (Experiments):
As for the experiments, I understand the proposed method is proposed for low-resource condition. However, the proposed method should be verified on popular tasks such as image generation (maybe the author can show that the proposed method can achieve a high performance with only 5% data compared with the conventional image generation methods). Currently the 3 task in paper is not quite convincing. And the time series task even has no previous published baselines (the baseline model is also proposed by the author).

Author response:
We did not apply our approach to domains such as image generation because those domains usually have huge labeled dataset (e.g., ImageNet, CelebFaces, COCO) that can be leveraged to directly train the diffusion model conditional on labels. Instead, we focus on modalities such as time series, molecules or motions that lack textual descriptions due to costly annotations or intricate data structures.

Reviewer comment (Experiments):
The authors claim that it is “usually easy to obtain by either manual collection of simulation” to get the unlabeled data. What’s the estimated manual data collecting effort and procedure? How do the authors design a system to do simulation? If data synthesis is based on an existing generative model, to train such a high-quality generative model already requires a lot of data.

Author response:
* Manual collection can be the existing unlabeled datasets such as QM9, ZINC, etc. Those datasets usually contains a huge amount of unlabeled samples to pretrain our model. For instance, QM9 has 134k stable organic molecules, and ZINC has 250k commercially available drug-like chemical compounds.

* Simulation can be, for example, simulating time series data from a statistical model, such as mixture autoregressive model [1], autoregressive moving-average model [2] etc. From those statistical models we can simulate as many as samples of time series, but they are still unlabeled.

[1] Kang et al., GeneRAting TIme Series with diverse and controllable characteristics.

[2] Benjamin et al., Generalized autoregressive moving average models.

Reviewer comment (Reproducibility):
The authors claim they use a cross-attention layer, but some implementation details are missing.

Author response:
We use the cross-attention to enforce the alignment between data and text descriptions. Specifically, we maximize the cosine-similarity between text embeddings obtained from the LLMs encoder and data embeddings diffused from original data. We have added the explanation to Appendix C.1 and the relevant part has been highlighted in red.

Reviewer comment (Experiments):
What’s the pre-training data? Do the authors use the same dataset of the downstream task for pre-training? By reading the 2nd paragraph of page 6, here’s my understanding: The authors identified N datasets with textual description. The authors create the synthesized ‘low-resource’ scenario by sampling varying proportions of the dataset. Pre-training is done on the full dataset for each concrete problem.

Author response:
Pre-training is done on the full training dataset, that involves both unlabeled and labeled data. Labeled data is treated as unlabeled without using their labels in the pre-training process. Finetuning relies on only those small proportion of labeled data in the training set.

Reviewer comment (Experiments):
The authors should have a baseline that trained on the 100% of the labeled data.

Author response:
We have added experiments for up to 100% of labeled data (e.g., 60%, 80%, 100%), the results of which are as below. The experiments are performed on QM9 dataset by leveraging EDM baseline.

| %   | Alpha     |              |        | HOMO      |              |        |
|-----|-----------|--------------|--------|-----------|--------------|--------|
|     | Text2Data | EDM-Finetune | EDM    | Text2Data | EDM-Finetune | EDM    |
| 2   | 0.6636    | 0.8378       | 0.8339 | 0.5792    | 0.9168       | 1.0049 |
| 4   | 0.6206    | 0.7446       | 0.6874 | 0.5583    | 0.8384       | 1.0708 |
| 6   | 0.4384    | 0.6783       | 0.6917 | 0.4364    | 0.9226       | 0.9996 |
| 8   | 0.5247    | 0.6687       | 0.6868 | 0.4658    | 0.9008       | 0.9477 |
| 10  | 0.3412    | 0.6588       | 0.7473 | 0.3299    | 0.8915       | 0.9156 |
| 20  | 0.4079    | 0.6004       | 0.6711 | 0.4241    | 0.8188       | 0.9134 |
| 40  | 0.4276    | 0.5546       | 0.5543 | 0.4238    | 0.7825       | 0.8050 |
| 60  | 0.4100    | 0.5011       | 0.5151 | 0.4300    | 0.6055       | 0.5912 |
| 80  | 0.4025    | 0.4050       | 0.4317 | 0.4217    | 0.5034       | 0.5120 |
| 100 | 0.4006    | 0.4019       | 0.3994 | 0.4189    | 0.4295       | 0.4234 |

Based on the results in the table above, both finetuned method without constraint (i.g., EDM-Finetune) and the direct conditional model (EDM) converge when the proportion goes up to 100%.

Reviewer comment (Presentation):
Please include proportions that are larger than 40 as reference.

Author response:
We have added experiments for up to 100% of labeled data (e.g., 60%, 80%, 100%), the results of which are as below. The experiments are performed on QM9 dataset by leveraging EDM baseline.

| %   | Alpha     |              |        | HOMO      |              |        |
|-----|-----------|--------------|--------|-----------|--------------|--------|
|     | Text2Data | EDM-Finetune | EDM    | Text2Data | EDM-Finetune | EDM    |
| 2   | 0.6636    | 0.8378       | 0.8339 | 0.5792    | 0.9168       | 1.0049 |
| 4   | 0.6206    | 0.7446       | 0.6874 | 0.5583    | 0.8384       | 1.0708 |
| 6   | 0.4384    | 0.6783       | 0.6917 | 0.4364    | 0.9226       | 0.9996 |
| 8   | 0.5247    | 0.6687       | 0.6868 | 0.4658    | 0.9008       | 0.9477 |
| 10  | 0.3412    | 0.6588       | 0.7473 | 0.3299    | 0.8915       | 0.9156 |
| 20  | 0.4079    | 0.6004       | 0.6711 | 0.4241    | 0.8188       | 0.9134 |
| 40  | 0.4276    | 0.5546       | 0.5543 | 0.4238    | 0.7825       | 0.8050 |
| 60  | 0.4100    | 0.5011       | 0.5151 | 0.4300    | 0.6055       | 0.5912 |
| 80  | 0.4025    | 0.4050       | 0.4317 | 0.4217    | 0.5034       | 0.5120 |
| 100 | 0.4006    | 0.4019       | 0.3994 | 0.4189    | 0.4295       | 0.4234 |

Based on the results in the table above, both finetuned method without constraint (i.g., EDM-Finetune) and the direct conditional model (EDM) converge when the proportion goes up to 100%.