Paper: Order-Preserving GFlowNets (ICLR-2024)

Reviewer comment (Experiments):
There can be more comparison against other state-of-the-art methods in the experimental section. The experiments, such as the hypergrid, only compares against the GFN-TB method. If the exploration-exploitation is the main benefit of OR-GFN in single objective problems, having more baseline comparison that encourages exploration of TB (such as https://arxiv.org/pdf/2306.17693.pdf or other variations of the reward besides beta-exponentiating) can make the paper stronger.

Author response:
**Exploration in TB**. Thanks for your insightful reference. We will add the paper to the related work.  We think the idea of Thompson sampling could be useful in OP-GFNs as well. However, one of the main benefits of our work is that it naturally balances the exploration and exploitation as the training goes, without the need for manually designed exploration steps. Specifically, compared with Thompson Sampling, sampling from $K$ different forward policies increases the computational budget for $\times K$. Please see the **Common Response Related Work**.

Reviewer comment (Reproducibility):
In single-objective maximization, the form of the local label y of the Pareto set is explicitly given in section 3.2, however in the multi-objective case, it is not clear how the label y can be calculated a priori, when the Pareto fronts are unknown. “When the true Pareto front is unknown, we use a discretization of the extreme faces of the objective space hypercube as P.” It is not clear whether the proposed method works due to the good estimate of the P, i.e., whether GGN can also benefit from incorporating estimated Pareto fronts into reward definition somehow.

Author response:
**Pareto front** Sorry for the confusion, the sentence "When the true Pareto front is unknown, we use a discretization of the extreme faces of the objective space hypercube as $P$", is used only in the evaluation instead of the training. When we do not know the true Pareto front, we can only use evenly spaced points on the outermost faces of the $[0,1]^D$ cube to measure the closeness or diversity of the generated samples. 
In the training, we describe how to label the samples in "Section 3.1, 1) Order preserving". In every step, we sample a batch of terminal states $X$ and label 1 to those which are non-dominated in $X$, and 0 otherwise. We will add this description to make it more clearly in the paper. Please note the local distribution in Section 3.2 is a special case of those in Section 3.1.

Reviewer comment (Experiments):
In section 4.2, the results shown are for the top-100 candidates, are the results similar for other choices of K?

Author response:
**Value of $k$ in top$k$** We think it might be unnecessary to compare many different $k$s. Since $k$ is only the moving window to reduce the possible variance in measuring the optimal candidates.

Reviewer comment (Novelty):
Related work on encouraging exploration of GFlowNets: https://arxiv.org/pdf/2306.17693.pdf

Author response:
**Exploration in TB**. Thanks for your insightful reference. We will add the paper to the related work.  We think the idea of Thompson sampling could be useful in OP-GFNs as well. However, one of the main benefits of our work is that it naturally balances the exploration and exploitation as the training goes, without the need for manually designed exploration steps. Specifically, compared with Thompson Sampling, sampling from $K$ different forward policies increases the computational budget for $\times K$. Please see the **Common Response Related Work**.

Reviewer comment (Writing):
Figure 4.1 caption, typo “Topk”

Author response:
**Minor: typos**. Thanks for pointing out the typos. We have corrected them in the updated manuscript.

Reviewer comment (Presentation):
Figure 4.1, should the legend be ‘TB-OP’ instead of ‘TB-RP’?

Author response:
**Minor: typos**. Thanks for pointing out the typos. We have corrected them in the updated manuscript.

Reviewer comment (Writing):
In section 4.1, typo “TB‘s performance”

Author response:
**Minor: typos**. Thanks for pointing out the typos. We have corrected them in the updated manuscript.

Reviewer comment (Reproducibility):
It's really not clear what the algorithm actually is; readers have to go all the way into the appendix to find it, and even there some questions remain, such as: are $\mathcal{T}$ and $\mathcal{D}$ distinct? What is $\hat R$ trained on? Does it have distinct parameters? Shared? etc.

Author response:
**Minor: $\mathcal{D}$ or $\mathcal{T}$**. Sorry for the confusion, $\mathcal{D}$ is a typo, it should indeed be $\mathcal{T}$ in Algorithm 1. We have corrected it in the updated manuscript.

**What $\widehat{R}$ is trained on?** $\widehat{R}$ is trained on the order-preserving loss $L_{OP} $, its parameters depend on GFN parametrization used, i.e., the parameters of: 1) $P_F$, $P_B$ and $Z$, 2) $P_F$, $P_B$ and $F$, or 3) $F$, as given in Table 2.1. parametrized by the GFN parametrization, such as $\widehat{R}_{\rm TB}(x;\theta)$.

Reviewer comment (Experiments):
From scrolling through the appendix, it appears that there are, understandably, a number of tricks that can be used. Most of them have some form of ablation in one task or the other, but a cohesive summary is lacking. The main criticism here is that it's important to disentangle what the contribution is from the algorithm itself, and from the tricks.

Author response:
**Disentangling algorithms with tricks**. Sorry for the confusion. We optionally use backward augmentation and backward KL regularization in the NAS environments, and clearly specify their usage by (-KL) and (-AUG) in the plots or tables. We have observed from Figure 4.2 that the major improvements are from the OP method, and these tricks only yield marginal improvements. More specifically, all 3 variations of OP-GFNs that include selections of KL regularization or training trajectory augmentation are very close to each other. In other environments, we did not include additional tricks when comparing with traditional GFNs but followed the same setup as the original papers. In other words, the only difference is the training criteria.

Reviewer comment (Presentation):
Investigative figures like Fig. E.6 should really be in the main body of the paper, and it would be much nicer to have such a figure for more complex domains. Generally the paper doesn't do a great job of showing _why_ the method works empirically.

Author response:
**Investigative figures in the appendix**. We admit some figures are left in the appendix. However, there is a 9-page limit, so we chose to put the ablation studies and auxiliary results in the appendix. We felt, that discussing the properties of the trained OP-GFN gives a good insight about what kind of generating distribution the OP-GFN learns. Hence, we sacrificed some space to include these theoretic results in the main paper and put the validation of the theoretical results in the appendix.

Reviewer comment (Writing):
For someone not familiar with GFNs, Eq (3) may not be obvious (R being parameterized "through" $P_F P_B$ & $Z$).

Author response:
**Descriptions of GFNs**. We succinctly describe the GFNs in Table 2.1. We have made a clearer reference to this table in the updated manuscript.

Reviewer comment (Novelty):
I'm not sure that I fully appreciate the "piecewise linear" part of the contribution. Is there something I'm missing?

Author response:
**Log piecewise linear**. We admit "log piecewise linear"  is not a good naming, and might be a confusing term. Following your arguments, we replace it with “piecewise geometric progression”. The result of "piecewise geometric progression" is not as trivial as it sounds, especially when there are candidates of the same objective value $u(\cdot)$, please refer Propositions 2 and 5 for details.

Reviewer comment (Theory):
The whole thing is discrete and operating on sets, so it makes no sense to talk about piecewise linearity because we are not in $\mathbb{R}$—the pieces "in between" don't even exist.

Author response:
**Log piecewise linear**. We admit "log piecewise linear"  is not a good naming, and might be a confusing term. Following your arguments, we replace it with “piecewise geometric progression”. The result of "piecewise geometric progression" is not as trivial as it sounds, especially when there are candidates of the same objective value $u(\cdot)$, please refer Propositions 2 and 5 for details.

Reviewer comment (Theory):
Proposition 3/6 is similarly weird—why this specific choice of MDP? I'd recommend either introducing why this choice matters, or maybe at least stating that the general case is much harder to provide theoretical statements for. I find the current form a bit awkward.

Author response:
**Sequence prepend/append MDP**. Sequence prepend/append MDP are widely used in molecule generation, such as in Section 4.2, “For the SIX6, PHO4, QM9, sEH environment, we use the sequence prepend/append MDP in Definition 1”. Therefore, theoretical analysis based on this MDP structure is meaningful.

Reviewer comment (Experiments):
In Section 4.2, the authors seem to be using the molecular setup of Shen et al., not Bengio et al. Shen et al.'s setup is quite different (and more simple—the state spaces are made to be much smaller so that computing $p(x; \theta)$ becomes tractable). It would be appropriate to note this. It's not clear what "choices" the authors are using from Shen et al. since quite a few are introduced in that work (which appear to have a significant impact).

Author response:
**Molecular setup**. We inherit the basic setup as Shen et al for trajectory balance, and for OP methods, we only change the training loss. Please see Section E.3.2, and **Common Response: Extensive Experiments**.

Reviewer comment (Experiments):
The "optionally use the backward KL regularization (-KL) and trajectories augmentation (-AUG) introduced in Appendix E.1" options don't seem to have that much impact, but it's a bit weird that they're just mentioned in passing in the main text.

Author response:
**Definition of -KL and -AUG**. We write the names in the main text because it appear in Figure 4.2, we defer the detailed description to the appendix due to the page limit.

Reviewer comment (Evaluation):
The statement "we observe that OP-GFNs can learn a highly sparse reward function that concentrates on the true Pareto solutions, outperforming PC-GFNs" is a bit weird because OP-GFNs and PC-GFNs learn entirely different things. From Fig 5.1 it seems like PC-GFN is attributing some probability mass on the Pareto front points, which suggests that it has discovered them.

Author response:
**PC-GFNs vs OP-GFNs in HyperGrid**. In Figure 5.1, PC-GFNs indeed attribute some probability mass on the Pareto front points, but not on all such points, such as "currin-shubert", it misses a lot of Pareto front solutions. Please see Figure F.1, this is mainly because "currin-shubert"'s Pareto front is concave, and linear scalarization in PC-GFNs inherently cannot deal with a concave Pareto front. We point out this weakness in Section 2.2. On the contrary, OP-GFNs can recover all the solutions, regardless of the Pareto front shape, and the landscape terminal flow is much more sparse than PC-GFNs.

Reviewer comment (Experiments):
This is maybe more of a comment on RL literature, but I'm also not a fan of comparing algorithms on a grid. Gridworlds are useful to sanity check algorithms, not to compare algorithms.

Author response:
**Sanity checking in HyperGrid**. Thanks for your advice on sanity checking in the grid-worlds. We perform the sanity checking in HyperGrid in the multi-objective optimization, since only in the hypergrid, the Pareto front can be explicitly calculated. We also perform extensive experiments on real-world environments, including molecular designs and NAS.

Reviewer comment (Writing):
In Sections 4 & 5, it would be good to clarify what work introduced what task and what MDP setup is being used.

Author response:
**Details in Section 4, 5**. We put the detailed setup in Sections 4 and 5 in the appendix due to the page limit. We hope the experiments' succinct description and reference to the appendix in the main text, as well as "Appendix A: an overview" could help you navigate the detailed settings.

Reviewer comment (Experiments):
There is a possibility of encountering non-stationarity issues when jointly training GFlowNets and the reward function. It might be worth exploring alternative training strategies to mitigate this potential challenge.

Author response:
1. **Non-stationarity issues**. We find the non-stationary issues are not very serious in the single-objective case. On the contrary, if we do not learn but define a reward in prior, like with GFN-$\beta$, we cannot obtain similar performance like with OP-GFNs. defining a reward in prior by GFN-$\beta$ methods are not as good as the OP-GFNs. Please see Appendix E.4.4. ablation studies. However,  In the multi-objective cases, we observe the non-stationary issues arise, and we choose to use the replay buffer to stabilize the training. Please see the second paragraph of Section 5.2. We added a sentence to make this more clear.

Reviewer comment (Experiments):
Experimental results are not that persuasive, having little improvement over baselines. For example, this work just compares with simple GFN baseline in molecular tasks, more competitive baselines (e.g., subTB, FL-GFN, RL methods) are needed.

Author response:
2. **More competitive baselines** From Figure 4.1, Figure 4.2, Figure 5.1, Figure F.1, our algorithm has proven to be more effective than previous methods. We have already include the subTB (Appendix E.4.5) and RL methods (REINFORCE, Figure 4.2) in NAS environments, which are, to our knowledge, strong baselines. Could you please elaborate in more detail, which baselines you would like to see in comparison? For the molecular design, please see the **Common Response: Extensive Experiments**.

Reviewer comment (Novelty):
There are some researches on temperature-conditioned GFlowNets, which learn a single model for multiple reward functions conditioned on different temperatures. It may be more persuasive to compare the proposed method with the literature [1], [2], especially in single-objective settings. While temperature-conditioned GFlowNets are too recently proposed to be included in this submission, it is suggested to include some discussion in the final version for future researchers.

Author response:
3. **Temperature conditioning methods** Please see the **Common Response: Related Work**.

Reviewer comment (Experiments):
In Section 4.2, authors say that they compare the proposed method with GFN-$\beta$, where $\beta$ is the selected value from the previous work. As far as known, the temperature is not tuned in the previous work. What if high values are assigned to $\beta$? It seems possible to achieve higher topk reward by sacrificing diversity when assigning high values to $\beta$, and authors only consider maximal objective in a single-experiment setting.

Author response:
4. **Setting higher $\beta$** Using GFN for stochastic optimization, (1) we do not only consider the maximal reward, but we also need to pay attention to the diversity (exploration). For example, in Section 4.2 (molecular design) we also measure the number of different optimal candidates. If the reward is very sparse, the sampler will only visit states with near-zero rewards, and cannot explore further to visit the various different maximal candidates. We validate this in Figure E.4, when setting $R_0=0$, it yields a very sparse reward landscape. As a result, and the sampler can only sample candidates near the initial points. (2) even if we only consider the maximal value regardless of the diversity, a large $\beta$ will still not be a good choice. Since a large $\beta$ might hinder the exploration to optimal candidates distant to the initialization. To point out this problem, we tested various For example, we test various $\beta$ in NAS in Figure E.8. Setting $\beta=128$ does not perform better than $\beta=16$.

Reviewer comment (Experiments):
In Sections 5.2 and 5.4, authors say they use a replay buffer and do off-policy training. There is curiosity as to whether, since the reward function is trained during training, off-policy training may lead to degrading performance.

Author response:
5. **Replay buffer** In the sparse reward landscape, off-policy training has been shown to help stabilize the training. Please refer to Appendix C.1 of the paper "Goal-conditioned GFlowNets for Controllable Multi-Objective Molecular Design" (https://arxiv.org/pdf/2306.04620.pdf). We add an additional sentence in Section 5.2 to state it more clearly.

Reviewer comment (Writing):
In Section 4.1, $u_0$ seems to be a typo. Maybe $R_0$ is intended.

Author response:
6. **Minor: $u_0$ or $R_0$** We always use $R_0$ to denote the constant in the objective of HyperGrid in the whole paper. Since either $u_0$ or $R_0$ merely represents a constant, we decide to use $R_0$ to align with the whole paper's notations, including the updated Figure E.5.

Reviewer comment (Reproducibility):
In Section 4.3, authors propose two implementation tricks: KL regularization and trajectories augmentation. Especially, trajectories augmentation has shown superior results in previous work. It is hard to capture when those two tricks are applied. While authors state that trajectory augmentation is applied in molecule design (E.3.2), the main paper says that trajectories augmentation is optionally used in the NAS environment. It may be helpful to clarify when those two tricks are applied across different experiment settings.

Author response:
7. **Minor: trajectory augmentation** We always use trajectory augmentation in molecule design as suggested in Appendix E.3.2. In the NAS environment, we optionally use trajectory augmentation, and use "-AUG" to if we use this augmentation in the algorithm, see Figure 4.2.

Reviewer comment (Experiments):
The experiments section can be expanded to include more difficult environments. For example, for hypergrid, higher values of H and N can be tested as larger grids will help analyzing the exploration problems better.

Author response:
**1. More complex environments**. In the submitted paper, we included many complex real-world environments apart from apart from HyperGrid, which includes molecular generation tasks and NAS. Such tasks are standard in evaluating GFNs in previous papers. We mostly use HyperGrid only for sanity-checking and therefore did not consider more complex configurations with larger $H$ and $D$ at first. To understand the exploration of the algorithm better, in the molecular design environments, we record the number of distinct optimal candidates. Please see the **Common Response: Extensive Experiments**. Furthermore, we additionally consider two high-dimensional settings in HyperGrid: $(D,H)=(3,32)$ and $(D,H)=(4,16)$, please see Section 4.1, and updated Figure E.5. We consider the following three ratios to measure exploration-exploitation: 1) \#(distinctly visited states)/\#(all the states); 2) \#(distinctly visited maximal states)/ \#(all the maximal states); 3) In the most recently 4000 visited states, \#(distinctly maximal states)/4000. A good sampling algorithm should have a small ratio 1), and a large ratio 2), 3).  We confirm that TB’s performance is sensitive to the choice of $R_0$, and observe that OP-TB can recover all maximal areas more efficiently, and sample maximal candidates with higher probability after visiting fewer distinct candidates.

Reviewer comment (Experiments):
Detailed balance objective can perform reasonably well in many settings. It will be beneficial to include it in all methods and numbers reported.

Author response:
**2. Including the detailed balance objective.** The submitted paper contains many experiments with detailed balance in the NAS environment in Appendix E.4.5, where we do not observe a significant gain from the trajectory balance. Therefore, we did not include detailed balance in other environments. However, we ran additional experiments and added detailed balance to the molecular design. Please see the **Common Response: Extensive Experiments**.

Reviewer comment (Evaluation):
It will be useful to add standard deviation and error bars across experiments. It will also be useful to better understand the variance across different GFlowNet objectives and baseline methods.

Author response:
**3. Adding error bars**. We agree with you and add the error bars to the experiments in the main paper, i.e., to the molecular design environments (Please see the **Common Response: Extensive Experiments**). We observe that the OP-TB's variance is small, and outperforms other methods beyond the error bars (especially in sehstr).

Reviewer comment (Experiments):
Is it possible to include some more difficult configurations of the environments? Hypergrid is one easy example for such an experiment.

Author response:
**1. More complex environments**. In the submitted paper, we included many complex real-world environments apart from apart from HyperGrid, which includes molecular generation tasks and NAS. Such tasks are standard in evaluating GFNs in previous papers. We mostly use HyperGrid only for sanity-checking and therefore did not consider more complex configurations with larger $H$ and $D$ at first. To understand the exploration of the algorithm better, in the molecular design environments, we record the number of distinct optimal candidates. Please see the **Common Response: Extensive Experiments**. Furthermore, we additionally consider two high-dimensional settings in HyperGrid: $(D,H)=(3,32)$ and $(D,H)=(4,16)$, please see Section 4.1, and updated Figure E.5. We consider the following three ratios to measure exploration-exploitation: 1) \#(distinctly visited states)/\#(all the states); 2) \#(distinctly visited maximal states)/ \#(all the maximal states); 3) In the most recently 4000 visited states, \#(distinctly maximal states)/4000. A good sampling algorithm should have a small ratio 1), and a large ratio 2), 3).  We confirm that TB’s performance is sensitive to the choice of $R_0$, and observe that OP-TB can recover all maximal areas more efficiently, and sample maximal candidates with higher probability after visiting fewer distinct candidates.

Reviewer comment (Evaluation):
It would be helpful if the variations across different seeds and runs could be provided for the experiments.

Author response:
**3. Adding error bars**. We agree with you and add the error bars to the experiments in the main paper, i.e., to the molecular design environments (Please see the **Common Response: Extensive Experiments**). We observe that the OP-TB's variance is small, and outperforms other methods beyond the error bars (especially in sehstr).

Reviewer comment (Experiments):
A hyperparam search for the best learning rate can also be sometimes a big contributing factor across different learning methods.

Author response:
**4. A hyperparameter search for the best learning rate**  We use the Adam optimizer for training criteria and parametrizations (DB, TB, subTB, OP). The Adam optimizer adaptively computes individual learning rates for
different parameters. Therefore, we did not finetune the learning rate of the Adam optimizer. For example, in the NAS environment, we use the initial learning rate $10^{-3}$ for network parameters and $10^{-1}$ for $Z$'s parameters.  
We think this comparison is fair because it shows that OP-GFN can outperform standard GFN by only replacing the training criterion. In other words, our results are not cherry-picked from the learning rate.