Paper: Compositional Generative Inverse Design (ICLR-2024)

Reviewer comment (Novelty):
The manuscript lacks discussion on some pivotal related works, notably the contributions by Ren et al. [1] and Ansari et al. [2]. Ren et al. elucidated various neural inversion methodologies and assessed their efficacy and accuracy encompassing Neural Adjoint, Tandem, Invertible Neural Networks, among other techniques. Additionally, they proposed a regularization scheme to mitigate the occurrence of out-of-distribution solutions.

Author response:
**Answer**: Thank you for the valuable feedback and the suggestion of the literature. In the updated manuscript, we have added in the “Related Work” section the citation to the paper "Benchmarking deep inverse models over time, and the neural-adjoint method" [1] and  "Autoinverse: Uncertainty aware inversion of neural networks" [2] as well as some related works "Design-bench: Benchmarks for data-driven offline model-based optimization" and "Bidirectional learning for offline infinite-width model-based optimization". While both of the literature [1, 2] perform inverse design for a wide range of applications, the literature has notable differences from our work (which makes it not feasible to compare within the experiment in the main text.) The methods in [1, 2] aim to avoid out-of-distribution solutions, while CinDM aims to generalize to out-of-distribution and more complex design inputs than seen in training. Neural Adjoint method in [1] relies on the boundary loss function defined by statistics of *training data* such as confidence interval and mean in the inverse design task, which in principle means once the design gets far from training data, the design variable is forced back to in-distribution domain. INN in [1, 2] also cannot generalize to a longer temporal scale because the temporal scale for the invertible neural network function is fixed. On the other hand, CinDM achieves generalization to **_out-of-distribution_** in terms of both spatial and temporal scales. The key idea that allows CinDM to generalize to out-of-distribution is to view the inverse design problem from an energy optimization perspective, where local constraints of the simulation model are implicitly captured through learning the generative energy function, and spatially and/or temporally compose multiple generative energy functions to govern global constraints. With this compositional feature, CinDM may design sequences of outputs that are significantly longer than the ones seen in training and design systems with many more objects and more complex shapes than those seen in training. Namely, our model is trained over trajectories with two bodies (in the N-body task) and a single boundary (in the 2D airfoils task) and is tested on larger numbers of bodies and boundaries for longer time horizons. Despite the difference, we acknowledge the importance of comparing our method to the representative baselines in [1, 2]. Thus we evaluated these baselines. The implementation and results are detailed in the response to the next comment.

Reviewer comment (Experiments):
These inversion methods should be mentioned and, where possible, compared with the proposed approach. In cases where a comparison is not possible, sufficient explanation is required.

Author response:
**Answer**: Thank you for the valuable feedback and the suggestion of the literature. In the updated manuscript, we have added in the “Related Work” section the citation to the paper "Benchmarking deep inverse models over time, and the neural-adjoint method" [1] and  "Autoinverse: Uncertainty aware inversion of neural networks" [2] as well as some related works "Design-bench: Benchmarks for data-driven offline model-based optimization" and "Bidirectional learning for offline infinite-width model-based optimization". While both of the literature [1, 2] perform inverse design for a wide range of applications, the literature has notable differences from our work (which makes it not feasible to compare within the experiment in the main text.) The methods in [1, 2] aim to avoid out-of-distribution solutions, while CinDM aims to generalize to out-of-distribution and more complex design inputs than seen in training. Neural Adjoint method in [1] relies on the boundary loss function defined by statistics of *training data* such as confidence interval and mean in the inverse design task, which in principle means once the design gets far from training data, the design variable is forced back to in-distribution domain. INN in [1, 2] also cannot generalize to a longer temporal scale because the temporal scale for the invertible neural network function is fixed. On the other hand, CinDM achieves generalization to **_out-of-distribution_** in terms of both spatial and temporal scales. The key idea that allows CinDM to generalize to out-of-distribution is to view the inverse design problem from an energy optimization perspective, where local constraints of the simulation model are implicitly captured through learning the generative energy function, and spatially and/or temporally compose multiple generative energy functions to govern global constraints. With this compositional feature, CinDM may design sequences of outputs that are significantly longer than the ones seen in training and design systems with many more objects and more complex shapes than those seen in training. Namely, our model is trained over trajectories with two bodies (in the N-body task) and a single boundary (in the 2D airfoils task) and is tested on larger numbers of bodies and boundaries for longer time horizons. Despite the difference, we acknowledge the importance of comparing our method to the representative baselines in [1, 2]. Thus we evaluated these baselines. The implementation and results are detailed in the response to the next comment.

Reviewer comment (Evaluation):
A proper discussion over the limitations is missing.

Author response:
**Answer**: In the original manuscript, we have already discussed the limitations of our method in Appendix G and referenced in the “Section 5. Conclusion” section. In the updated manuscript, the discussions are improved and are presented in the Appendix J, which we elaborate below:

(1) In terms of design quality and exploration space, we need to strike a balance between different objectives to avoid getting stuck in local optima, especially when dealing with complex, nonlinear systems in the real world. We also need to ensure that the designed samples adhere to complex multi-scale physical constraints.

(2) Furthermore, achieving interpretability in the samples designed by deep learning models is challenging for inverse design applications.

(3) From a cost perspective, training diffusion models requires large datasets and intensive computational resources. The complexity of calculations also hinders the speed of our model design.

Reviewer comment (Reproducibility):
The code and dataset is missing.

Author response:
**Answer**: We release our code to reviewers via the anonymous link: https://anonymous.4open.science/r/CinDM_anonymous-0CAF, inside which we also present downloading link to the example datasets and trained model checkpoints in README.md file. The current datasets contain the full 2-body dataset and part of the 2D airfoils dataset. We have in total 30,000 airfoil trajectories, which is about 400GB, and we release 1,00 trajectories among them due to space limitations. Reviewers can follow the instructions to install the environment and run the code on the example datasets. Our code will be public once our paper is accepted.

Reviewer comment (Experiments):
It is unclear how other neural inversion methods perform in the context of the proposed experiments.

Author response:
**Answer**: Thank you for the suggestions. In the updated manuscript, we have evaluated two additional baselines: neural adjoint method + boundary loss function (NABL) and conditional invertible neural network (cINN) method for both N-body and 2D airfoils experiments (they are added to Appendix H). Since it has been shown in  [1] that cINN outperforms INN, we adopt cINN instead of INN for comparison. We note that tandem in [1] is unfeasible in our problem settings because the output dimension is smaller than the dimension of the input of our models.

We implement NABL on top of our previous baseline architectures, namely FNO and LE-PDE in the 2D airfoils task and U-Net in the N-body task, named “NABL, FNO”, “NABL, LE-PDE” and “NABL, U-Net”, respectively. These new NABL baselines additionally use the boundary loss defined by the mean value and 95% significance radius of the training dataset [1], which aims to mitigate the occurrence of out-of-distribution solutions.

For cINN, we use the implementation from [https://github.com/BensonRen/BDIMNNA](https://github.com/BensonRen/BDIMNNA), which is suggested by [1]. cINN is not applicable to compositional design because the input scale for the invertible neural network function is fixed. Therefore, in the N-body time composition task, we trained 4 cINN models, each for one of the time steps: 24, 34, 44, and 54. These models differ only in the input size. The input $x$ to cINN is a vector of size  2 * 4 * number_steps. The condition y is set to 0, the minimal distance to the target point. For cINN for 2D airfoil design, we adopt as the input 2D coordinates of 40 boundary points, which are spanned into an 80-dimensional vector, since the invertible constraint on the cINN model hardly accepts image-like inputs adopted in the main experiments. Therefore we evaluate cINN only in the single airfoil design task. The condition $y$ is set as the minimal value of drag - lift drag in the training trajectories. In both tasks, the random variable $z$ has a dimension of dim($x$) - dim($y$). It is drawn from a Gaussian distribution and then input to the INN for inference. We also adjust the hyperparameters, such as hidden size and the number of reversible blocks, to make the number of parameters in cINN close to ours for a fair comparison. The results of NABL and cINN are shown in Table 10 and Table 11 below. Due to limitation of the number of words, we present analysis of results in the next section of response.

Reviewer comment (Experiments):
The sensitivity of this inversion method to hyperparameters and initialization is not addressed.

Author response:
**Answer**:

The key hyperparameter in our method is $\lambda$, a tradeoff between the (compositional) energy model and design objective. Its influence is also concerned by Reviewer 4Mfs and Reviewer 7GzX. We have performed additional experiments to test the influence of $\lambda$ for both N-body and 2D airfoils inverse design. The results are listed in the following Table 12 and Table 13. From these tables, we can see that our method is robust and has steady performance in a wide range of $\lambda$. If $\lambda$ is set too small (<=0.0001 in the 2D airfoils task, or <0.01 in the N-body task), then the design results are inferior because only little objective guidance is added. If $\lambda$ is set too large (>0.01 in the 2D airfoils task, or >1.0 in the N-body task), then it is prone to enter a poor likelihood region, and physical consistency is not well preserved. In practice, $\lambda$ could be set as 0.01 to 1.0 for the N-body task and 0.0002 to 0.02 for the 2D airfoils task.

The evaluation results and analysis of $\lambda$ are presented in Appendix I.1 in the updated manuscript.

Table 12. Effect of $\lambda$ in 2 airfoils inverse design.

| $\lambda$ | obj | lift/drag |
| --- | --- | --- |
| 0.05 | 0.7628±0.1892 | 1.0150±0.2008 |
| 0.02 | 0.3849±0.0632 | 1.0794±0.1165 |
| 0.01 | 0.2292±0.0408 | 1.2860±0.1402 |
| 0.005 | 0.2061±0.0388 | 1.2378±0.1414 |
| 0.002 | 0.2170±0.0427 | 1.2429±0.1243 |
| 0.001 | 0.2277±0.0451 | 1.2608±0.1469 |
| 0.0005 | 0.2465±0.0473 | 1.4102±0.1771 |
| **0.0002** | **0.1986±0.0431** | **1.4216±0.1607** |
| 0.0001 | 0.2710±0.0577 | 1.1962±0.1284 |

Table 13. Effect of $\lambda$ in N-body time composition inverse design.

| $\lambda$ | design_obj(2-body 24 steps) | MAE(2-body 24 steps) | design_obj(2-body 34 steps) | MAE(2-body 34 steps) | design_obj(2-body 44 steps) | MAE(2-body 44 steps) | design_obj(2-body 54 steps) | MAE(2-body 54 steps) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0.0001 | 0.3032 ± 0.0243 | 0.00269 ± 0.00047 | 0.2954 ± 0.0212 | 0.00413 ± 0.00155 | 0.3091 ± 0.0223 | 0.00394 ± 0.00076 | 0.2996 ± 0.0201 | 0.01046 ± 0.00859 |
| 0.001 | 0.2531 ± 0.0185 | 0.00385 ± 0.00183 | 0.2937 ± 0.0213 | 0.00336 ± 0.00115 | 0.2797 ± 0.0190 | 0.00412 ± 0.00105 | 0.2927 ± 0.0219 | 0.00521 ± 0.00103 |
| 0.01 | 0.1200 ± 0.0069 | 0.00483 ± 0.00096 | 0.1535 ± 0.0135 | 0.00435 ± 0.00100 | 0.1624 ± 0.0137 | 0.00416 ± 0.00059 | 0.1734 ± 0.0154 | 0.00658 ± 0.00267 |
| 0.1 | 0.1201 ± 0.0046 | 0.01173 ± 0.00150 | 0.1340 ± 0.0107 | 0.00772 ± 0.00099 | 0.1379 ± 0.0088 | 0.00816 ± 0.00149 | 0.1662 ± 0.0180 | 0.01141 ± 0.00473 |
| 0.2 | 0.1283 ± 0.0141 | 0.01313 ± 0.00312 | 0.1392 ± 0.0119 | 0.00836 ± 0.00216 | 0.1529 ± 0.0130 | 0.01019 ± 0.00584 | 0.1513 ± 0.0131 | 0.00801 ± 0.00172 |
| **0.4** | **0.1143 ± 0.0047** | **0.01202 ± 0.00114** | **0.1251 ± 0.0071** | **0.00763 ± 0.00069** | **0.1326 ± 0.0087** | **0.00695 ± 0.00067** | **0.1533 ± 0.0140** | **0.00870 ± 0.00150** |
| 0.6 | 0.1259 ± 0.0100 | 0.01382 ± 0.00115 | 0.1326 ± 0.0126 | 0.01171 ± 0.00595 | 0.1592 ± 0.0151 | 0.01140 ± 0.00355 | 0.1670 ± 0.0177 | 0.00991 ± 0.00287 |
| 0.8 | 0.1217 ± 0.0073 | 0.01596 ± 0.00127 | 0.1385 ± 0.0120 | 0.01095 ± 0.00337 | 0.1573 ± 0.0116 | 0.00893 ± 0.00113 | 0.1715 ± 0.0181 | 0.01026 ± 0.00239 |
| 1 | 0.1330 ± 0.0063 | 0.01679 ± 0.00139 | 0.1428 ± 0.0112 | 0.01087 ± 0.00149 | 0.1634 ± 0.0119 | 0.00968 ± 0.00079 | 0.1789 ± 0.0164 | 0.01102 ± 0.00185 |
| 2 | 0.1513 ± 0.0079 | 0.02654 ± 0.00160 | 0.1795 ± 0.0129 | 0.01765 ± 0.00193 | 0.1779 ± 0.0121 | 0.01707 ± 0.00474 | 0.2113 ± 0.0161 | 0.01447 ± 0.00130 |
| 10 | 0.2821 ± 0.0197 | 0.21153 ± 0.01037 | 0.2210 ± 0.0149 | 0.09715 ± 0.00236 | 0.2273 ± 0.0133 | 0.07781 ± 0.00232 | 0.2269 ± 0.0175 | 0.06538 ± 0.00210 |

Due to limitation of the number of words, we present analysis of sensitivity of initialization in the next section of response.

Reviewer comment (Reproducibility):
maybe I missed it, but page 7, I don't think $M$ is ever defined. How exactly do you train $M$ beyond the range of timesteps in training?

Author response:
**Answer**: $M$ is exactly the same as $N$ in Eq. (7), which means the number of time trunks for time composition. Actually, M is a hyperparameter and could not be “trained”. By using $M \geq 2$, our model could generalize to more time steps than training data in inverse design. In the experiment, we tested the performance of time composition on $M=1, 2, 3, 4$ for the N-body inverse design task. The results are shown in Table 1 of the manuscript.

We replaced M with N to avoid such confusion in the updated manuscript.

Reviewer comment (Theory):
I would question the compositionality of the method and call it a "piecewise" or "mixture" approach? Given that you simply partition the spaces required into overlapping pieces (unless I misunderstood something)

Author response:
**Answer**: Although our method looks like a “mixture” approach, there is a substantial difference. Because the energy function can be viewed as the (unnormalized) negative log-probability by the relation $E_{\theta}(x) \propto -\log p(x)$, our summation of energy functions of several components in compositional design corresponds to taking products of probability of each design component, which is an approximation of the joint distribution of the involved components. Thus the compositional energy function essentially models the *****_***concurrence***_***** of those components in a probabilistic approach, which could naturally govern their physics. It is quite different to a simple mixture over partitioned space with overlapping pieces, which lacks an effective global physical constraint in the whole space and thus may produce design results that fail to preserve physical consistency across subspaces. Our compositional energy approach is similar to past work [1] and it has been shown to be an effective way for compositional tasks. We hope our explanation could address the reviewer's concern.

[1] Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fer- gus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc. arXiv preprint arXiv:2302.11552, 2023.

Reviewer comment (Experiments):
Were the numbers of parameters matched for the different baselines? Given that you a partitioned energy functions, there might be potential for unfairness here?

Author response:
**Answer**: Thanks for your reminder of the model size. This is an important problem in fair comparison. We count the number of parameters in all the baselines and our method. The results are shown in the following Table 6 and Table 7.

For the airfoils inverse design task, we see from Table 6 that the model size of CinDM is between the two baselines FNO and LEPDE. To make a fair comparison, we adjust the model size of all the baselines to align with our CinDM. “CEM, FNO” and “Backprop, FNO” share the same model architecture, so do “CEM, LE-PDE” and “Backprop, LE-PDE”. Note that we could not make the model size of baselines exactly the same as ours because the number of parameters is determined by hyperparameters of model architectures, i.e. hidden layer numbers and latent embedding size. Baselines with the updated number of parameters are trained and evaluated again. Furthermore, to make the simulation more accurate and convincing, we use a 128 x 128 resolution of flow field, instead of 64 x 64 as we did in the initial submission, for evaluation of the 2D airfoils design task. The new results are shown in the following Table 8. The results confirm that our CinDM still outperforms all the baselines significantly regarding lift/drag metric under comparable model size.

For the N-body inverse design task, Table 7 reveals that CinDM and U-Net have comparable model sizes, both larger than GNS. The small number of parameters in GNS is due to its utilization of a semi-implicit Euler integration for updating node features, leveraging existing physical prior knowledge. Consequently, GNS does not require a large number of parameters. GNS restricts its input features solely to the current time step, making it ineffective to employ a larger model.

Table 6. Comparison of the number of parameters for different methods in 2D airfoils inverse design task.

| model | #Parameters (million) |
| --- | --- |
| FNO | 7.39 |
| LE-PDE | 1.83 |
| CinDM | 3.11 |

Table 7. Comparison of the number of parameters for different methods in N-body inverse design task.

| model | #Parameters (million) (23-steps) | #Parameters (million) (1-step) |
| --- | --- | --- |
| GNS | 0.22 | 0.21 |
| U-Net | 20.45 | 19.88 |
| CinDM | 20.76 | 20.19 |

Table 8. Comparison of CinDM with baselines under comparable model size for the 2D airfoils inverse design task.


| Methods          | #Parameters(million) | obj(1 airfoil) | lift/drag(1 airfoil) | obj(2 airfoils) | lift/drag(2 airfoils) |
|------------------|----------------------|:--------------:|:--------------------:|:--------------:|:--------------------:|
| CEM, FNO         |         3.29         |     0.0932     |        1.4005        |     0.3890     |        1.0914        |
| CEM, LE-PDE      |         3.13         |     0.0794     |        1.4340        |     0.1691     |        1.0568        |
| Backprop, FNO    |         3.29         |     0.0281     |        1.3300        |     0.1837     |        0.9722        |
| Backprop, LE-PDE |         3.13         |     0.1072     |        1.3203        |     0.0891     |        0.9866        |
| CinDM            |         3.11         |     0.0797     |         2.177        |     0.1986     |        1.4216        |

The new results on comparison with baselines under the comparable number of parameters for the 2D airfoils inverse design task are presented in Table 3 of Section 4.3 in the updated manuscript.

Reviewer comment (Experiments):
Did you try training a single shared network across the overlapping chunks? I was kind of expecting something like this (maybe with different degrees of subsampling to give long and short range dynamics)

Author response:
**Answer**: Thanks for suggesting such an important baseline. To address this question, we train two diffusion models with 44 steps for the N-body task, aiming to perform inverse design directly without time composition. Their parameter sizes are roughly equivalent to CinDM and twice as large.  All other training configurations are kept identical to those used in CinDM. The results are presented in the following Table 9. It clearly shows the new baseline design results are not superior to those achieved by CinDM. Due to the long-range dependencies among different time steps, capturing the features of trajectories across 44 steps simultaneously using a single model is challenging. In this case, a 24-step diffusion model is more suitable. Therefore, for designs involving more time steps, employing time composition is proven to be a more effective approach, with lower cost and better performance.

Table 9. Comparison to the baseline: directly diffuse 44 steps without time composition in an N-body inverse design task.

| Methods                   | #Parameters(million) | design_obj(2-body 44 steps) | MAE(2-body 44 steps) | design_obj(4-body 44 steps) | MAE(4-body 44 steps) |
|---------------------------|:--------------------:|-----------------------------|----------------------|-----------------------------|----------------------|
| Our method                | 20.76M               | 0.1326 ± 0.0087             | 0.00695 ± 0.00067    | 0.2281 ± 0.0145           | 0.03195 ± 0.00705    |
| Directly diffuse 44 steps | 44.92M               | 0.2779 ± 0.0197             |  0.00810 ± 0.00200   | 0.2986 ± 0.0148            | 0.05166 ± 0.01218    |

The evaluation results and analysis of this new baseline for N-body time composition inverse design task are presented in Appendix C in the updated manuscript.

Reviewer comment (Experiments):
This is more of a question. On the joint optimization, it is trying to minimize the energy component which is calculated from the trajectories and the boundary, and minimizing the design objective as well. It is proposed to achieve this by optimizing the design and the trajectory at the same time. In the joint optimization formulation as in Eqn.(3), the design objective function is weighted by $\lambda$. I am curious how this hyperparameter is estimated/configured, and how sensitive the optimization results are to the change in $\lambda$.

Author response:
We thank the reviewer for the positive feedback. We appreciate that the reviewer thinks our work is novel in method, complete in experiments, and promising in more applications. Below, we address the reviewer’s concern on the hyperparameter $\lambda$.

>__Comment 1__:  I am curious how this hyperparameter (lambda) is estimated/configured, and how sensitive the optimization results are to the change in (lambda).

**Answer**: This is a very important question for the evaluation performance of our method. It is also concerned by Reviewer 4Mfs and Reviewer WWyF. We have performed additional experiments to test the influence of $\lambda$ for both N-body and 2D airfoils inverse design. The results are listed in the following Table 4 and Table 5. From these tables, we can see that our method is robust and has steady performance in a wide range of $\lambda$. If $\lambda$ is set too small ($\leq 0.0001$ in the 2D airfoils task, or $< 0.01$ in the N-body task), then the design results are inferior because only little objective guidance is added. If $\lambda$ is set too large ($> 0.01$ in the 2D airfoils task, or $> 1.0$ in the N-body task), then it is prone to enter a poor likelihood region, and physical consistency is not well preserved. In practice, $\lambda$ could be set as 0.01 to 1.0 for the N-body task and 0.0002 to 0.02 for the 2D airfoils task. In our paper, we choose $\lambda$ based on the best evaluation performance, namely we set $\lambda$ as 0.4 for the N-body task and 0.0002 for the 2D airfoils task.

The evaluation results and analysis of $\lambda$ are presented in Appendix I.1 in the updated manuscript.

Table 4. Effect of $\lambda$ in 2 airfoils inverse design.

| $\lambda$ | obj | lift/drag |
| --- | --- | --- |
| 0.05 | 0.7628±0.1892 | 1.0150±0.2008 |
| 0.02 | 0.3849±0.0632 | 1.0794±0.1165 |
| 0.01 | 0.2292±0.0408 | 1.2860±0.1402 |
| 0.005 | 0.2061±0.0388 | 1.2378±0.1414 |
| 0.002 | 0.2170±0.0427 | 1.2429±0.1243 |
| 0.001 | 0.2277±0.0451 | 1.2608±0.1469 |
| 0.0005 | 0.2465±0.0473 | 1.4102±0.1771 |
| **0.0002** | **0.1986±0.0431** | **1.4216±0.1607** |
| 0.0001 | 0.2710±0.0577 | 1.1962±0.1284 |

Table 5. Effect of $\lambda$ in N-body time composition inverse design.

| $\lambda$ | design_obj(2-body 24 steps) | MAE(2-body 24 steps) | design_obj(2-body 34 steps) | MAE(2-body 34 steps) | design_obj(2-body 44 steps) | MAE(2-body 44 steps) | design_obj(2-body 54 steps) | MAE(2-body 54 steps) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0.0001 | 0.3032 ± 0.0243 | 0.00269 ± 0.00047 | 0.2954 ± 0.0212 | 0.00413 ± 0.00155 | 0.3091 ± 0.0223 | 0.00394 ± 0.00076 | 0.2996 ± 0.0201 | 0.01046 ± 0.00859 |
| 0.001 | 0.2531 ± 0.0185 | 0.00385 ± 0.00183 | 0.2937 ± 0.0213 | 0.00336 ± 0.00115 | 0.2797 ± 0.0190 | 0.00412 ± 0.00105 | 0.2927 ± 0.0219 | 0.00521 ± 0.00103 |
| 0.01 | 0.1200 ± 0.0069 | 0.00483 ± 0.00096 | 0.1535 ± 0.0135 | 0.00435 ± 0.00100 | 0.1624 ± 0.0137 | 0.00416 ± 0.00059 | 0.1734 ± 0.0154 | 0.00658 ± 0.00267 |
| 0.1 | 0.1201 ± 0.0046 | 0.01173 ± 0.00150 | 0.1340 ± 0.0107 | 0.00772 ± 0.00099 | 0.1379 ± 0.0088 | 0.00816 ± 0.00149 | 0.1662 ± 0.0180 | 0.01141 ± 0.00473 |
| 0.2 | 0.1283 ± 0.0141 | 0.01313 ± 0.00312 | 0.1392 ± 0.0119 | 0.00836 ± 0.00216 | 0.1529 ± 0.0130 | 0.01019 ± 0.00584 | 0.1513 ± 0.0131 | 0.00801 ± 0.00172 |
| **0.4** | **0.1143 ± 0.0047** | **0.01202 ± 0.00114** | **0.1251 ± 0.0071** | **0.00763 ± 0.00069** | **0.1326 ± 0.0087** | **0.00695 ± 0.00067** | **0.1533 ± 0.0140** | **0.00870 ± 0.00150** |
| 0.6 | 0.1259 ± 0.0100 | 0.01382 ± 0.00115 | 0.1326 ± 0.0126 | 0.01171 ± 0.00595 | 0.1592 ± 0.0151 | 0.01140 ± 0.00355 | 0.1670 ± 0.0177 | 0.00991 ± 0.00287 |
| 0.8 | 0.1217 ± 0.0073 | 0.01596 ± 0.00127 | 0.1385 ± 0.0120 | 0.01095 ± 0.00337 | 0.1573 ± 0.0116 | 0.00893 ± 0.00113 | 0.1715 ± 0.0181 | 0.01026 ± 0.00239 |
| 1 | 0.1330 ± 0.0063 | 0.01679 ± 0.00139 | 0.1428 ± 0.0112 | 0.01087 ± 0.00149 | 0.1634 ± 0.0119 | 0.00968 ± 0.00079 | 0.1789 ± 0.0164 | 0.01102 ± 0.00185 |
| 2 | 0.1513 ± 0.0079 | 0.02654 ± 0.00160 | 0.1795 ± 0.0129 | 0.01765 ± 0.00193 | 0.1779 ± 0.0121 | 0.01707 ± 0.00474 | 0.2113 ± 0.0161 | 0.01447 ± 0.00130 |
| 10 | 0.2821 ± 0.0197 | 0.21153 ± 0.01037 | 0.2210 ± 0.0149 | 0.09715 ± 0.00236 | 0.2273 ± 0.0133 | 0.07781 ± 0.00232 | 0.2269 ± 0.0175 | 0.06538 ± 0.00210 |

Reviewer comment (Experiments):
Additionally, I wonder whether changing its value will lead to different results in the experiments.

Author response:
We thank the reviewer for the positive feedback. We appreciate that the reviewer thinks our work is novel in method, complete in experiments, and promising in more applications. Below, we address the reviewer’s concern on the hyperparameter $\lambda$.

>__Comment 1__:  I am curious how this hyperparameter (lambda) is estimated/configured, and how sensitive the optimization results are to the change in (lambda).

**Answer**: This is a very important question for the evaluation performance of our method. It is also concerned by Reviewer 4Mfs and Reviewer WWyF. We have performed additional experiments to test the influence of $\lambda$ for both N-body and 2D airfoils inverse design. The results are listed in the following Table 4 and Table 5. From these tables, we can see that our method is robust and has steady performance in a wide range of $\lambda$. If $\lambda$ is set too small ($\leq 0.0001$ in the 2D airfoils task, or $< 0.01$ in the N-body task), then the design results are inferior because only little objective guidance is added. If $\lambda$ is set too large ($> 0.01$ in the 2D airfoils task, or $> 1.0$ in the N-body task), then it is prone to enter a poor likelihood region, and physical consistency is not well preserved. In practice, $\lambda$ could be set as 0.01 to 1.0 for the N-body task and 0.0002 to 0.02 for the 2D airfoils task. In our paper, we choose $\lambda$ based on the best evaluation performance, namely we set $\lambda$ as 0.4 for the N-body task and 0.0002 for the 2D airfoils task.

The evaluation results and analysis of $\lambda$ are presented in Appendix I.1 in the updated manuscript.

Table 4. Effect of $\lambda$ in 2 airfoils inverse design.

| $\lambda$ | obj | lift/drag |
| --- | --- | --- |
| 0.05 | 0.7628±0.1892 | 1.0150±0.2008 |
| 0.02 | 0.3849±0.0632 | 1.0794±0.1165 |
| 0.01 | 0.2292±0.0408 | 1.2860±0.1402 |
| 0.005 | 0.2061±0.0388 | 1.2378±0.1414 |
| 0.002 | 0.2170±0.0427 | 1.2429±0.1243 |
| 0.001 | 0.2277±0.0451 | 1.2608±0.1469 |
| 0.0005 | 0.2465±0.0473 | 1.4102±0.1771 |
| **0.0002** | **0.1986±0.0431** | **1.4216±0.1607** |
| 0.0001 | 0.2710±0.0577 | 1.1962±0.1284 |

Table 5. Effect of $\lambda$ in N-body time composition inverse design.

| $\lambda$ | design_obj(2-body 24 steps) | MAE(2-body 24 steps) | design_obj(2-body 34 steps) | MAE(2-body 34 steps) | design_obj(2-body 44 steps) | MAE(2-body 44 steps) | design_obj(2-body 54 steps) | MAE(2-body 54 steps) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0.0001 | 0.3032 ± 0.0243 | 0.00269 ± 0.00047 | 0.2954 ± 0.0212 | 0.00413 ± 0.00155 | 0.3091 ± 0.0223 | 0.00394 ± 0.00076 | 0.2996 ± 0.0201 | 0.01046 ± 0.00859 |
| 0.001 | 0.2531 ± 0.0185 | 0.00385 ± 0.00183 | 0.2937 ± 0.0213 | 0.00336 ± 0.00115 | 0.2797 ± 0.0190 | 0.00412 ± 0.00105 | 0.2927 ± 0.0219 | 0.00521 ± 0.00103 |
| 0.01 | 0.1200 ± 0.0069 | 0.00483 ± 0.00096 | 0.1535 ± 0.0135 | 0.00435 ± 0.00100 | 0.1624 ± 0.0137 | 0.00416 ± 0.00059 | 0.1734 ± 0.0154 | 0.00658 ± 0.00267 |
| 0.1 | 0.1201 ± 0.0046 | 0.01173 ± 0.00150 | 0.1340 ± 0.0107 | 0.00772 ± 0.00099 | 0.1379 ± 0.0088 | 0.00816 ± 0.00149 | 0.1662 ± 0.0180 | 0.01141 ± 0.00473 |
| 0.2 | 0.1283 ± 0.0141 | 0.01313 ± 0.00312 | 0.1392 ± 0.0119 | 0.00836 ± 0.00216 | 0.1529 ± 0.0130 | 0.01019 ± 0.00584 | 0.1513 ± 0.0131 | 0.00801 ± 0.00172 |
| **0.4** | **0.1143 ± 0.0047** | **0.01202 ± 0.00114** | **0.1251 ± 0.0071** | **0.00763 ± 0.00069** | **0.1326 ± 0.0087** | **0.00695 ± 0.00067** | **0.1533 ± 0.0140** | **0.00870 ± 0.00150** |
| 0.6 | 0.1259 ± 0.0100 | 0.01382 ± 0.00115 | 0.1326 ± 0.0126 | 0.01171 ± 0.00595 | 0.1592 ± 0.0151 | 0.01140 ± 0.00355 | 0.1670 ± 0.0177 | 0.00991 ± 0.00287 |
| 0.8 | 0.1217 ± 0.0073 | 0.01596 ± 0.00127 | 0.1385 ± 0.0120 | 0.01095 ± 0.00337 | 0.1573 ± 0.0116 | 0.00893 ± 0.00113 | 0.1715 ± 0.0181 | 0.01026 ± 0.00239 |
| 1 | 0.1330 ± 0.0063 | 0.01679 ± 0.00139 | 0.1428 ± 0.0112 | 0.01087 ± 0.00149 | 0.1634 ± 0.0119 | 0.00968 ± 0.00079 | 0.1789 ± 0.0164 | 0.01102 ± 0.00185 |
| 2 | 0.1513 ± 0.0079 | 0.02654 ± 0.00160 | 0.1795 ± 0.0129 | 0.01765 ± 0.00193 | 0.1779 ± 0.0121 | 0.01707 ± 0.00474 | 0.2113 ± 0.0161 | 0.01447 ± 0.00130 |
| 10 | 0.2821 ± 0.0197 | 0.21153 ± 0.01037 | 0.2210 ± 0.0149 | 0.09715 ± 0.00236 | 0.2273 ± 0.0133 | 0.07781 ± 0.00232 | 0.2269 ± 0.0175 | 0.06538 ± 0.00210 |

Reviewer comment (Reproducibility):
Can the authors describe the role of $\alpha$ line 12, Alg.1 ?

Author response:
**Answer**: This is a typo. $\alpha$ should be removed.

It is fixed in the updated version.

Reviewer comment (Evaluation):
Can the authors comment on the choice of the energy function for the airfold design ? How do we compare to training data ?

Author response:
**Answer**: The energy function is used to model the joint distribution of trajectory and boundary data in our physical evolution. In airfoil design, it is very hard to explicitly compute this energy function due to high dimensional spatial-temporal dependency and complex gas-solid coupling. So we take an implicit way to characterize the energy function: we use a stochastic process to learn the gradient of the energy function from observed data and take an opposite direction to reach the low energy region during inference. By introducing a design objective term, we can generate samples that achieve better objective values than training data while maintaining physical consistency as training data. Furthermore, this implicit energy function makes it easy for compositional design, which could generate samples that are substantially different from training data, like the compositional design of formation flying of 2 airfoils. We think it is really challenging to think of other choices of energy function for airfoil design without considering stochastic processes, especially when one wants compositional airfoil design.

Reviewer comment (Experiments):
Can the authors comment on how to balance $\lambda$ during the optimization ? Could the optimization end up in a poor likelihood region ?

Author response:
**Answer**: We have performed additional experiments to test the influence of $\lambda$ for both N-body and 2D airfoils inverse design. The results are listed in the following Table 1 and Table 2. From these tables, we can see that our method is robust and has steady performance in a wide range of $\lambda$. If $\lambda$ is set too small ($\le 0.0001$ in the 2D airfoils task, or $<0.01$ in the N-body task), then the design results are inferior because only little objective guidance is added. If $\lambda$ is set too large ($>0.01$ in the 2D airfoils task, or $>1.0$ in the N-body task), then it is prone to fall into a poor likelihood region, and physical consistency is not well preserved. In practice, $\lambda$ could be set as 0.01 to 1.0 for the N-body task and 0.0002 to 0.02 for the 2D airfoils task. In our paper, we choose $\lambda$ based on the best evaluation performance, namely we set $\lambda$ as 0.4 for the N-body task and 0.0002 for the 2D airfoils task.

For a forward/optimization strategy, there are some other choices. In forward, our network predicts the noise of each step, perhaps we can instead predict $z_0$ in Eq. (6) of Section 3.2, which contains clean boundary and state variables, in each forward step. This approach has been shown to be effective in some generation tasks. In inference, we can use optimization techniques like DDIM [1] to speed up the procedure of generation. For details about the evaluation of DDIM on our method, please refer to our response to the next question.

The evaluation results and analysis of $\lambda$ are presented in Appendix I.1 in the updated manuscript.

[1] Song, J., Meng, C., & Ermon, S. (2020). Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502.

Table 1. Effect of $\lambda$ in 2 airfoils inverse design.

| $\lambda$ | obj | lift/drag |
| --- | --- | --- |
| 0.05 | 0.7628±0.1892 | 1.0150±0.2008 |
| 0.02 | 0.3849±0.0632 | 1.0794±0.1165 |
| 0.01 | 0.2292±0.0408 | 1.2860±0.1402 |
| 0.005 | 0.2061±0.0388 | 1.2378±0.1414 |
| 0.002 | 0.2170±0.0427 | 1.2429±0.1243 |
| 0.001 | 0.2277±0.0451 | 1.2608±0.1469 |
| 0.0005 | 0.2465±0.0473 | 1.4102±0.1771 |
| **0.0002** | **0.1986±0.0431** | **1.4216±0.1607** |
| 0.0001 | 0.2710±0.0577 | 1.1962±0.1284 |

Table 2. Effect of $\lambda$ in N-body time composition inverse design.

| $\lambda$ | design_obj(2-body 24 steps) | MAE(2-body 24 steps) | design_obj(2-body 34 steps) | MAE(2-body 34 steps) | design_obj(2-body 44 steps) | MAE(2-body 44 steps) | design_obj(2-body 54 steps) | MAE(2-body 54 steps) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0.0001 | 0.3032 ± 0.0243 | 0.00269 ± 0.00047 | 0.2954 ± 0.0212 | 0.00413 ± 0.00155 | 0.3091 ± 0.0223 | 0.00394 ± 0.00076 | 0.2996 ± 0.0201 | 0.01046 ± 0.00859 |
| 0.001 | 0.2531 ± 0.0185 | 0.00385 ± 0.00183 | 0.2937 ± 0.0213 | 0.00336 ± 0.00115 | 0.2797 ± 0.0190 | 0.00412 ± 0.00105 | 0.2927 ± 0.0219 | 0.00521 ± 0.00103 |
| 0.01 | 0.1200 ± 0.0069 | 0.00483 ± 0.00096 | 0.1535 ± 0.0135 | 0.00435 ± 0.00100 | 0.1624 ± 0.0137 | 0.00416 ± 0.00059 | 0.1734 ± 0.0154 | 0.00658 ± 0.00267 |
| 0.1 | 0.1201 ± 0.0046 | 0.01173 ± 0.00150 | 0.1340 ± 0.0107 | 0.00772 ± 0.00099 | 0.1379 ± 0.0088 | 0.00816 ± 0.00149 | 0.1662 ± 0.0180 | 0.01141 ± 0.00473 |
| 0.2 | 0.1283 ± 0.0141 | 0.01313 ± 0.00312 | 0.1392 ± 0.0119 | 0.00836 ± 0.00216 | 0.1529 ± 0.0130 | 0.01019 ± 0.00584 | 0.1513 ± 0.0131 | 0.00801 ± 0.00172 |
| **0.4** | **0.1143 ± 0.0047** | **0.01202 ± 0.00114** | **0.1251 ± 0.0071** | **0.00763 ± 0.00069** | **0.1326 ± 0.0087** | **0.00695 ± 0.00067** | **0.1533 ± 0.0140** | **0.00870 ± 0.00150** |
| 0.6 | 0.1259 ± 0.0100 | 0.01382 ± 0.00115 | 0.1326 ± 0.0126 | 0.01171 ± 0.00595 | 0.1592 ± 0.0151 | 0.01140 ± 0.00355 | 0.1670 ± 0.0177 | 0.00991 ± 0.00287 |
| 0.8 | 0.1217 ± 0.0073 | 0.01596 ± 0.00127 | 0.1385 ± 0.0120 | 0.01095 ± 0.00337 | 0.1573 ± 0.0116 | 0.00893 ± 0.00113 | 0.1715 ± 0.0181 | 0.01026 ± 0.00239 |
| 1 | 0.1330 ± 0.0063 | 0.01679 ± 0.00139 | 0.1428 ± 0.0112 | 0.01087 ± 0.00149 | 0.1634 ± 0.0119 | 0.00968 ± 0.00079 | 0.1789 ± 0.0164 | 0.01102 ± 0.00185 |
| 2 | 0.1513 ± 0.0079 | 0.02654 ± 0.00160 | 0.1795 ± 0.0129 | 0.01765 ± 0.00193 | 0.1779 ± 0.0121 | 0.01707 ± 0.00474 | 0.2113 ± 0.0161 | 0.01447 ± 0.00130 |
| 10 | 0.2821 ± 0.0197 | 0.21153 ± 0.01037 | 0.2210 ± 0.0149 | 0.09715 ± 0.00236 | 0.2273 ± 0.0133 | 0.07781 ± 0.00232 | 0.2269 ± 0.0175 | 0.06538 ± 0.00210 |

Reviewer comment (Experiments):
What is the influence of the number of optimization steps ?

Author response:
**Answer**: We use DDIM sampling methods to evaluate the effect of different optimization steps on the design performance of the N-body task of our methods. As the results shown in Table 3, which are visualized in Fig 16 and Fig 17 of Appendix I.3, it is evident that an augmentation in the number of sampling time steps corresponds to a gradual reduction in the design objective. Conversely, MAE exhibits fluctuations within a narrow range, occasionally experiencing increments.

This observed phenomenon can be rationalized as follows: With an increase in the number of sampling steps, the design objective becomes intricately involved in the diffusion process. Consequently, the designs improve and closely align with the design objective, resulting in a decrease in the design objective value. However, as the number of sampling steps increases, the MAE also rises. This is attributed to a concentration of trajectories within a narrow range when there are a small number of sampling steps, causing some designed samples to have very low initial velocities. As a result, both the true trajectory and the diffused trajectory become highly concentrated, leading to a small calculated MAE value. Through a meticulous analysis of the sensitivity of the design objective and MAE to different sampling steps, we can confidently assert that CinDM achieves desired design outcomes aligned with the design objectives and physical constraints. This is accomplished by judiciously selecting an appropriate sampling step size during the inverse design process.

The evaluation results and analysis of the number of optimization steps in inference are presented in Appendix I.3 in the updated manuscript.

Table 3 The performance of different optimization (sample) steps in N-body time composition inverse design of CinDM.

| Sample steps | design obj | MAE |
| --- | --- | --- |
| 50 | 0.2339 | 0.00429 |
| 100 | 0.1778 | 0.00338 |
| 150 | 0.1767 | 0.00408 |
| 200 | 0.1510 | 0.00465 |
| 250 | 0.1468 | 0.00648 |
| 300 | 0.1359 | 0.00676 |
| 350 | 0.1321 | 0.00766 |
| 400 | 0.1235 | 0.00710 |
| 450 | 0.1329 | 0.00862 |
| 500 | 0.1192 | 0.00738 |
| 550 | 0.1185 | 0.01060 |
| 600 | 0.1250 | 0.01195 |
| 650 | 0.1273 | 0.01323 |
| 700 | 0.1159 | 0.01041 |
| 750 | 0.1176 | 0.01117 |
| 800 | 0.1217 | 0.01221 |
| 850 | 0.1193 | 0.01043 |
| 900 | 0.1179 | 0.01236 |
| 950 | 0.1189 | 0.01350 |
| 1000 | 0.1143 | 0.01224 |

Reviewer comment (Evaluation):
For the airfold design: what is the relationship between the initial objective function and the reported ratio ? Which quantity is actually at stake here ?

Author response:
**Answer**: Lift-to-drag ratio (https://www.sciencedirect.com/topics/engineering/lift-to-drag-ratio) is a standard evaluation metric in aerodynamics. We adopt the objective function, instead of lift/drag, as guidance during inference because it performs more steadily than lift/drag during optimization. Therefore, lift-to-drag ratio is a more practical metric and more attention should be paid when evaluating design results. We also show the objective as a metric because it is an indicator of the optimization performance.

Reviewer comment (Experiments):
Can the authors think of any limitation when applying a compositional energy approach ? For instance is it computationally efficient to learn “smaller models” vs one big EBM ?

Author response:
**Answer**: It is indeed more computationally efficient to learn individual components since they are simpler functions. However, the composition may not fully accurately capture the full distribution as it assumes independence in different components. The advantage of the compositional energy approach is that it provides an implicit way to model the energy landscape of global physical states, thus producing design results closer to physical reality on compositional tasks.