Paper: GFLOWNET TRAINING BY POLICY GRADIENTS (ICLR-2024)

Reviewer comment (Writing):
There are many instances of unclear and incorrect mathematical language and notation in sections 2 and 3.

Author response:
1. Unclear and incorrect mathematical language and notation in section 2 and 3:
   * Section 2.1: 
      * Yes, we meant that the DAG is graded and that the layers are indexed by $0,\ldots,T$. We have rephrased the sentence as ''Assuming that there is a topological ordering $S_1,\ldots,S_T$ for $T$ disjoint subsets of $S$''. The definition of the topological ordering follows Koller & Friedman (2009)[1].
      *  We follow definition 1 of  Bengio et al. (2021b)[2], where a partial order, is a homogeneous relation on a set that is reflexive, antisymmetric, and transitive (Wallis,2011)[3].
      * We have replaced the definitions as requirements. 
   * Section 2.2:
      * We correct this as ''$F$ is a measure''.
      * We add the missing condition in equation (2).
      * We add the definition of the backward policy below equation (1).
      * In terms of distributions, $\hat{Z}$ is the normalizing constant. Supposing a categorical distribution $P(\cdot)$ for $K$ kinds of states with parameters $(\theta_0,\ldots,\theta_k,\ldots,\theta_K)$, $P(\cdot)$ should be defined as $P(s^k):=\frac{\theta_k}{\hat{Z}}$ with $\hat{Z}=\sum_{k=0}^K\theta_k$, so that $\sum_{k=0}^K P(s^k)=1$. In terms of gradient computation, $\hat{Z}$ is considered constant w.r.t $\theta_{1:K}$, so $\nabla P(s^k|\theta_{0:K})=\frac{\nabla \theta_k}{\hat{Z}}$ . Distribution $\mu(s^0)=\frac{Z}{\hat{Z}}$ corresponds to a categorical distribution with only one category and one parameter $Z$. Thus, $\hat{Z}=Z$, which we call ''$\hat{Z}$ is clamped to $Z$". In terms of the gradient computation, $\nabla\mu(s^0;Z)=\frac{\nabla Z}{\hat{Z}}$. 
   * The beginning of Section 3: 
      * We thank the reviewer for pointing out the wrong citation, which has been corrected. Zimmermann et al. (2022)[4] do talk about the relationship between TB and KL. Their main contributions are KL-based objectives, which are shown experimentally to be effective objectives for GFN training. Nevertheless, the authors do not establish the relationship between TB and KL. So we put the citation to this paper in the `Related Work' Section. 
   * Section 3.1:
      *  We have rephrased the sentence as "the backward gradient equivalence requires computing the expectation over $P_B^\top(x)=R(x)/Z^*$, which is impossible". What we mean here is that: to build a practical training method following the equivalence by Malkin et al. (2022)[5], we need to do sampling from $P_B^\top(x)=R(x)/Z^\ast$. Sampling from $R(x)/Z^\ast$ is impossible by our assumption and this is just what GFlowNet is designed for. However, sampling from $P_B(\tau|x)$ conditioning on $x$ is tractable since $P_B(\tau|x)=\prod_{t=1}^{T}P_B(s_{t-1}|s_{t})$ ($s_T=x$) and $P_B(\cdot|\cdot)$ is the backward sampling policy. Thus our proposition 1 can imply a practical training method over $P(B|x)\rho(x)$.
      * We acknowledge that Malkin et al. (2022)[5] did consider gradients w.r.t $b:=logZ$ but did not establish the equivalence between TB and KL w.r.t $b$ in equations (7) and (8) of this paper.  Equation (12) does not complete the equivalence. It shows that: when you compute the gradients of the TB objective w.r.t $b$,  $\nabla_{b}L_{TB}(b)=E_{P_F(\tau)}\left[\nabla_{b}\left(c(\tau)+b\right)^2\right]=2E_{P_F(\tau)}\left[\left(c(\tau)+b\right)\right]$ ($c(\tau):=log\frac{P_F(\tau)}{R(x)P_B(\tau|x)}$); then  
gradient-based updating rules of $b$ with the learning rate $\frac{\eta}{2}$ is 
$b'\leftarrow 
 b-\eta E_{P_F(\tau)}\left[\left(c(\tau)+b\right)\right]=(1-\eta)b-\eta E_{P_F(\tau)}\left[c(\tau)\right]=(1-\eta)b+\eta b_{local}$, which is the equation (12).  We can see that no equivalence between TB and KL w.r.t $logZ$ was established during the whole derivation.  In light of this and our answer above about "sampling from $P_B$', we disagree with the comments that `nothing original in Proposition 1".
      * We have moved the remark below Proposition 1 in the revised manuscript.

Reviewer comment (Theory):
In section 2.1, the statement "Assume there are $T+1$ topological orderings" is incorrect; that is not what "topological ordering" means. A topological ordering is a partial order homomorphism from a given partial order to a total order (i.e., a list of the states where no state is preceded by its descendant). It is unclear whether the intended meaning is that the DAG is graded (as stated in the following paragraph) and that the layers are indexed by $0,\dots,T$.

Author response:
* Section 2.1: 
      * Yes, we meant that the DAG is graded and that the layers are indexed by $0,\ldots,T$. We have rephrased the sentence as ''Assuming that there is a topological ordering $S_1,\ldots,S_T$ for $T$ disjoint subsets of $S$''. The definition of the topological ordering follows Koller & Friedman (2009)[1].

Reviewer comment (Theory):
In section 2.1, the text incorrectly treats an instance of two elements being in a binary relation as a "partial order." A partial order is a set with a binary relation (satisfying some properties), equivalent to a DAG.

Author response:
* Section 2.1: 
      *  We follow definition 1 of  Bengio et al. (2021b)[2], where a partial order, is a homogeneous relation on a set that is reflexive, antisymmetric, and transitive (Wallis,2011)[3].

Reviewer comment (Theory):
In section 2.1, it is incorrect to **define** ${\cal S}_0:=\{s_0\}$ and similarly for ${\cal S}_T$ if the stratification into ${\cal S}_t$ already comes with the data of a graded DAG. The fact that $s^0$ is the only state in the first graded component ${\cal S}_0$ is an **assumption/requirement**, not a definition.

Author response:
* Section 2.1: 
      * We have replaced the definitions as requirements.

Reviewer comment (Theory):
In section 2.2, the second sentence is self-contradictory. A "probability measure" means the measure of the entire $\cal T$ is 1. In fact, $F$ is just a measure, and $P$, its normalization, is a probability measure.

Author response:
* Section 2.2:
      * We correct this as ''$F$ is a measure''.

Reviewer comment (Theory):
Equation (2) in section 2.2 is missing a condition that it should hold only for $s \neq s^0$.

Author response:
* Section 2.2:
      * We add the missing condition in equation (2).

Reviewer comment (Writing):
In equation (3) of section 2.2, the backward policy is used but was not introduced or defined.

Author response:
* Section 2.2:
      * We add the definition of the backward policy below equation (1).

Reviewer comment (Writing):
In section 2.2, the meaning of the statement "$\hat Z$ is a constant whose value is clamped to $Z$" is unclear.

Author response:
* Section 2.2:
      * In terms of distributions, $\hat{Z}$ is the normalizing constant. Supposing a categorical distribution $P(\cdot)$ for $K$ kinds of states with parameters $(\theta_0,\ldots,\theta_k,\ldots,\theta_K)$, $P(\cdot)$ should be defined as $P(s^k):=\frac{\theta_k}{\hat{Z}}$ with $\hat{Z}=\sum_{k=0}^K\theta_k$, so that $\sum_{k=0}^K P(s^k)=1$. In terms of gradient computation, $\hat{Z}$ is considered constant w.r.t $\theta_{1:K}$, so $\nabla P(s^k|\theta_{0:K})=\frac{\nabla \theta_k}{\hat{Z}}$ . Distribution $\mu(s^0)=\frac{Z}{\hat{Z}}$ corresponds to a categorical distribution with only one category and one parameter $Z$. Thus, $\hat{Z}=Z$, which we call ''$\hat{Z}$ is clamped to $Z$". In terms of the gradient computation, $\nabla\mu(s^0;Z)=\frac{\nabla Z}{\hat{Z}}$.

Reviewer comment (Novelty):
At the start of section 3, the claim that [Madan et al., 2023] discusses the relationship between TB and KL divergence is doubtful. This relationship is instead discussed in [Malkin et al., 2022b] and [Zimmermann et al., "A variational perspective on generative flow networks", TMLR].

Author response:
* The beginning of Section 3: 
      * We thank the reviewer for pointing out the wrong citation, which has been corrected. Zimmermann et al. (2022)[4] do talk about the relationship between TB and KL. Their main contributions are KL-based objectives, which are shown experimentally to be effective objectives for GFN training. Nevertheless, the authors do not establish the relationship between TB and KL. So we put the citation to this paper in the `Related Work' Section.

Reviewer comment (Theory):
In section 3.1, the claim "Expectation over $P_B$ is intractable due to unknown $Z^*$" is incorrect; even with known $Z^*$, sampling $P_B$ is not necessarily tractable.

Author response:
* Section 3.1:
      *  We have rephrased the sentence as "the backward gradient equivalence requires computing the expectation over $P_B^\top(x)=R(x)/Z^*$, which is impossible". What we mean here is that: to build a practical training method following the equivalence by Malkin et al. (2022)[5], we need to do sampling from $P_B^\top(x)=R(x)/Z^\ast$. Sampling from $R(x)/Z^\ast$ is impossible by our assumption and this is just what GFlowNet is designed for. However, sampling from $P_B(\tau|x)$ conditioning on $x$ is tractable since $P_B(\tau|x)=\prod_{t=1}^{T}P_B(s_{t-1}|s_{t})$ ($s_T=x$) and $P_B(\cdot|\cdot)$ is the backward sampling policy. Thus our proposition 1 can imply a practical training method over $P(B|x)\rho(x)$.

Reviewer comment (Novelty):
In section 3.1, the claim "This forward gradient equivalence [in [Malkin et al., 2022b]] does not take the total flow estimator into account" is incorrect. That paper shows that the equivalence on $P_F$ parameter gradients holds independently of $Z$ and also states the equivalence for the $\log Z$ gradient (after equation (12) in that paper), implying there is **nothing original in Proposition 1**.

Author response:
* Section 3.1:
      * We acknowledge that Malkin et al. (2022)[5] did consider gradients w.r.t $b:=logZ$ but did not establish the equivalence between TB and KL w.r.t $b$ in equations (7) and (8) of this paper.  Equation (12) does not complete the equivalence. It shows that: when you compute the gradients of the TB objective w.r.t $b$,  $\nabla_{b}L_{TB}(b)=E_{P_F(\tau)}\left[\nabla_{b}\left(c(\tau)+b\right)^2\right]=2E_{P_F(\tau)}\left[\left(c(\tau)+b\right)\right]$ ($c(\tau):=log\frac{P_F(\tau)}{R(x)P_B(\tau|x)}$); then  
gradient-based updating rules of $b$ with the learning rate $\frac{\eta}{2}$ is 
$b'\leftarrow 
 b-\eta E_{P_F(\tau)}\left[\left(c(\tau)+b\right)\right]=(1-\eta)b-\eta E_{P_F(\tau)}\left[c(\tau)\right]=(1-\eta)b+\eta b_{local}$, which is the equation (12).  We can see that no equivalence between TB and KL w.r.t $logZ$ was established during the whole derivation.  In light of this and our answer above about "sampling from $P_B$', we disagree with the comments that `nothing original in Proposition 1".

Reviewer comment (Theory):
In Proposition 1, it should be made clear that gradients are not propagated to the distribution $P_F$ over which the expectation is taken. Without this clarification, the proposition is not true. This remark is made in Appendix A but is not referenced in the main text.

Author response:
* Section 3.1:
      * We have moved the remark below Proposition 1 in the revised manuscript.

Reviewer comment (Writing):
The paragraphs following Definition 1 are extremely difficult to understand, as they switch between GFlowNets and regular RL settings.

Author response:
2. The paragraphs following Definition 1 are extremely difficult to understand:
    * We have significantly revised the text. We acknowledge that the MDPs are either ergodic or absorbing.  What we mainly convey here is that: under the absorbing MDP specified by a DAG, functions $V$ and $Q$ can be time-invariant; by contrast, they have to be time-varying in general cases since the graph formed by all possible paths can contain cycles.

Reviewer comment (Theory):
The statement in these paragraphs that in "regular RL" the Markov chain determined by the policy and transition environment is ergodic is objectionable, as many RL settings feature "irreversible" actions.

Author response:
2. The paragraphs following Definition 1 are extremely difficult to understand:
    * We have significantly revised the text. We acknowledge that the MDPs are either ergodic or absorbing.  What we mainly convey here is that: under the absorbing MDP specified by a DAG, functions $V$ and $Q$ can be time-invariant; by contrast, they have to be time-varying in general cases since the graph formed by all possible paths can contain cycles.

Reviewer comment (Theory):
The math following Definition 1 is related to that in max-ent RL, which effectively places negative log-policy into the reward.

Author response:
* In Section 7.2 of Bengio et al. (2021b)[2], the authors talked about the relationship between 
    Maximum Entropy (MaxEnt) RL and GFlowNets. The main message is that MaxEnt RL can be understood as learning a policy such that the terminating state distribution $P^\top(x)\propto n(x)R(x)$, where $n(x)$ is the number of paths in the DAG of all trajectories that lead to $x$, which is fundamentally different from the task of GFlowNets to learn a policy such that $P^\top(x)\propto R(x)$. This fact is also the main motivation of the GFlowNet model as explained in Proposition 1 of Bengio et al. (2021a)[6].

Reviewer comment (Theory):
It is unclear how the policy gradient methods described after Definition 1 can work off policy, which is an important advantage of GFlowNets and important for preventing mode collapse.

Author response:
*  Policy-based RL can be run in off-policy ways (e.g. by importance sampling (Degris et al., 2012)[7]) and can also utilize temperature conditioning, but it is beyond the scope of our work.

Reviewer comment (Theory):
The second sentence of section 3.3, which seems key, does not make sense. The backward policy is a distribution over parents of each state and induces a distribution over subtrajectories. It is unclear how $P_B$ could be higher or lower for trajectories that precede a high-reward state $x$ if it is conditioned on $x$.

Author response:
3. The second sentence of Section 3.3  seems key but does not make sense: 
    * We suggest referring to the sections "Compositional reward functions" and "The substructure credit assignment problem" by Shen et al. (2023)[8]. We are not conditioning on $x$ for sub-trajectories. First, $P_B(\cdot|\cdot)$ is the backward policy, and by definition, the backward distribution of a sub-trajectory $\bar{\tau}$ that ends in $s_n$ (i.e conditioning on $s_n$) is $P_B(\bar{\tau}=(s_m\rightarrow \ldots\rightarrow s_n)|s_n)=\prod_{t=m+1}^n P_B(s_{t-1}|s_{t})$.  Now let's suppose that $\bar{\tau}\in \tau|x=(s^0\rightarrow\ldots s_m\rightarrow \ldots s_n\rightarrow,\ldots x\rightarrow s^f)$ and $R(x)$ is high. Now if we want the probability of sampling $\tau$ that ends in $x$ to be high, we need to make the probability of sampling $\bar{\tau}$ to be high.

Reviewer comment (Writing):
The grammar in the second sentence of section 3.3 should be checked.

Author response:
*   We have corrected the grammar error.

Reviewer comment (Experiments):
The experiment results are not very convincing because, on the hypergrid, the policy-based methods effectively learn from partial trajectory information. A fairer comparison would be to subtrajectory balance, which performs better on large hypergrids ([Madan et al.]).

Author response:
4. The experiment results are not very convincing:
     * Yes, sub-trajectory objectives may help improve performance. But how to properly split a complete trajectory in the multiple sub-trajectories and how to combine the responding sub-trajectory objectives so that the overall performance can be improved is a tricky engineering problem. As our work focuses on connecting GFlowNet training to RL,  we do not consider sub-trajectory distribution following the choices of Malkin et al. (2022)[5] and Shen et al. (2023)[8], but our RL methods do apply to sub-trajectories as shown in Proposition 3 in Appendix A.3 and we provide additional results under the Detailed Balance Objective. (The detailed Balance objective corresponds to sub-trajectories with length 1, and TB corresponds to complete trajectories. )

Reviewer comment (Experiments):
In section 4.2 and 4.3, RL-G is not tested, and the reason for this omission is not explained.

Author response:
* We add the missing results under RL-G for other experiments.

Reviewer comment (Reproducibility):
It is unclear how training hyperparameters were selected.

Author response:
* We add the detail about the hyper-parameter setup in the appendix.

Reviewer comment (Evaluation):
The evaluation of total variation in sections 4.2 and 4.3 is questionable if it is performed by sampling, which can induce high variance in the estimate. It should be possible to compute the true terminating distribution, at least for the structure learning problem. JSD between the full distributions would allow comparison with past work; without this, there is no way to validate that TB faithfully reproduces results from [Deleu et al.] and [Malkin et al., 2022b].

Author response:
* We use dynamical programming to compute the learned terminating distribution and update the results when the space scale is relatively small. We add the performance results measured by JSD.

Reviewer comment (Writing):
There are issues with citation formatting (`\citet`/`\citep`) and inconsistent capitalization in "GFlowNet," which is incorrect in multiple places.

Author response:
5. Miscellaneous:
    * We thank the reviewer for pointing out these typos. We have corrected the citation typos and capitalization in ``GFlowNet".

Reviewer comment (Presentation):
There is a broken reference at the bottom of page 5.

Author response:
5. Miscellaneous:
    * We have fixed the broken reference.

Reviewer comment (Presentation):
The presentation of section 3 could be heavily reworked to be more concise and tell a story, making it clear at every step what the symbols are, where they come from, and how they contribute to progressing from TB to RL with a non-stationary reward. While the authors’ intent is somewhat understandable, the current presentation requires effort (such as consulting the appendix) to grasp.

Author response:
*  Presentation: 
     * We rewrite Section 3 as suggested to make every step of derivations clearer.

Reviewer comment (Experiments):
The empirical results are weak due to the use of relatively easy and uninteresting problems like hypergrids and bit sequences, which were originally introduced as toy examples. While learning to produce Bayesian nets is interesting, the authors do not properly compare their work to Deleu et al.

Author response:
*  Results:
    * We re-conduct the Bayesian Network structure learning experiment with additional results under the Detailed Balance objectives.
    * We have added more explanations about `why it is good' in the experiment section.
* We understand your concerns that some people bring in unnecessary tools to solve problems that are trivial or have been solved. However, Bengio et al. (2021a)[1] do not actually consider the problems in reinforcement learning and we bridge their work to reinforcement learning by defining the proper value function and other RL components, and consequently to further improve the performances.  As shown in the hyper-grid experiment, our new RL-formulated method can solve the problem not only more quickly (convergence rate), but also much more scalable to a larger scale than being reported in Bengio et al. (2021a)[1] and Malkin et al. (2022)[2]. Therefore, We think our work can serve as a stepping stone to make GFlowNets useful in real-world applications. Besides,  we redesigned the bit-sequence experiment as bio-sequence experiments based on real-world datasets to further validate our proposed methods.

Reviewer comment (Evaluation):
The paper lacks an analysis of “why does this work [better]?”, which is typically an important element to include in such research.

Author response:
*  Results:
    * We have added more explanations about `why it is good' in the experiment section.

Reviewer comment (Theory):
It seems far-fetched to classify this as a policy gradient algorithm, or possibly even RL, because the reward changes at every state in a self-referential manner. Similar methods, like Random Network Distillation, decouple the update of the reward from the policy update. Merely having an update equation that resembles PG does not necessarily make this classification scientifically useful. Moreover, methods like TRPO assume a fixed reward, raising uncertainty about the correctness of the described approach.

Author response:
* Yes, we agree that TRPO assumes that the reward is fixed.  In light of this fact, we introduce Theorem 2 as the theoretical guarantee with the non-stationary reward and absorbing MDP.  Besides, we add more explanations in Appendix B.4 about the stability of our methods under non-stationary rewards.

Reviewer comment (Theory):
The reviewer questions the statement: "Value-based methods, however, face the difficulties of designing a powerful sampler that can balance the exploration and exploitation trade-off, especially when the combinatorial space is enormous with well-isolated modes," noting they do not see how this challenge is specific to value-based methods. Soft-Q-Learning and SubTB-GFlowNets handle this trade-off effectively.

Author response:
* Yes, for regular RL problems, soft-Q learning can balance the trade-off.  However, soft-Q learning relies on the extended Bellman equation that incorporates a KL divergence as the regularized term. In terms of GFlowNets, it means that a KL regularized term should be incorporated into the flow balance equation, which is not straightforward. We do acknowledge that it is a good research direction. Actually, in Section 7.2 of Bengio et al. (2021b)[3], the authors talked about the relationship between GFlowNet and Maximization Entropy (MaxEnt) RL, and soft-Q learning is one of the representative MaxEnt RL methods. It pointed out that MaxEnt RL can be understood as learning a policy such that the terminating state distribution $P^{\top}(x)\propto n(x)R(x)$, where $n(x)$ is the number of paths in the DAG of all trajectories that lead to $x$, which is different from the task in GFlowNets to learn a policy such that $P^{\top}(x)\propto R(x)$. This fact is also the main motivation of the GFlowNet model as explained in Proposition 1 of Bengio et al. (2021a)[1]. By contrast, our work formally bridges the "flow" in GFlowNets to RL by defining non-stationary rewards, allowing us to perform the same tasks as GFlowNet in RL formulations.  
    * Sub-trajectory objectives may help improve performance. But how to properly split a complete trajectory in the multiple sub-trajectories and how to combine the responding sub-trajectory objectives so that the overall performance can be improved is a tricky engineering problem. As our work focuses on connecting GFlowNet training to RL, we do not consider sub-trajectory distribution following the choices of Malkin et al. (2022)[2] and Shen et al. (2023)[4], but our RL methods do apply to sub-trajectories as shown in Proposition 3 in Appendix A.3. Besides, temperature conditioning corresponds to using an off-policy sampler, which is different from the forward policy during training. Policy-based RL can be run in off-policy ways (e.g. by importance sampling (Degris et al., 2012)[5]) and can also utilize temperature conditioning, but it is beyond the scope of our work.

Reviewer comment (Writing):
The reviewer does not understand if the authors mean something specific or different when discussing absorbing MDPs, since GFlowNets already operate on absorbing DAGs via the definition of \(s_f\).

Author response:
* We correct the statement as $G$ already specifies an absorbing MDP. What we want to convey here is that for a normal absorbing MDP, functions $V$ and $Q$ have to be time-dependent in that the graph formed by all possible paths can contain cycles. Here we want to emphasize the fact that when the graph contains no cycle, functions $V$ and $Q$ can be time-independent, giving rise to modeling convenience.

Reviewer comment (Theory):
In section 3.2, when the authors write "It is clear that...", the reviewer disagrees and suggests it would be helpful to explain the derivation of the equation and why \(V_F\) can be substituted for (4).

Author response:
* We rewrite Section 3 and add explanations in Appendix B.4 further clarifying Why $V_F$ can be substituted for (4).

Reviewer comment (Theory):
The statement “Following the pipeline by Shen et al. (2023), we must solve the optimization of LTB−G to find the desired \(P_B\) at first” appears inaccurate, as in Shen et al., \(P_B\) and \(P_F\) are trained simultaneously, with \(P_B\) being non-parametric and evolving over time as more states are visited.

Author response:
* In the main text after Theorem 6.1 in Shen et al. (2023)[4], the authors stated that ''We propose to first train PB to match $p(\tau|x)$'' and ''Once converged, we freeze PB, then learn PF with fixed PB using trajectory balance, which recovers a proper GFlowNet learning objective''. Thus, in principle, their framework should be done in two phases. We acknowledge that in the following `Training Consideration' section, the authors mixed $P_B$ and $P_G$ by $\alpha P_B+(1-\alpha)P_G$ in the training objective w.r.t $P_F$, avoiding doing two-phase training in practice. This operation, however, lacks theoretical guarantees as the mixed distribution is still non-Markovian. By comparison, we provide a theoretical guarantee of our coupled training strategy.

Reviewer comment (Experiments):
For hypergrids, while gridworlds are useful to sanity check an idea, they should not be considered a strong basis for comparing algorithms.

Author response:
* Hypergrid: 
   * Following Shen et al. (2023)[4], we resign the bit-sequence experiment as bio-sequence experiments based on real-world datasets (including nucleotide strings and molecule graphs).

Reviewer comment (Theory):
For hypergrids, the claim that \( [P] \) can be estimated empirically by sampling is incomplete, since it can also be computed exactly through dynamic programming.

Author response:
* We update the performance results computed by dynamic programming in the hyper-grid experiment.

Reviewer comment (Experiments):
In the BN experiments, Deleu et al. use detailed balance, not TB, to train their model. For a fair comparison, performance should be compared against their exact setup rather than a potentially suboptimal one.

Author response:
*  Results:
    * We re-conduct the Bayesian Network structure learning experiment with additional results under the Detailed Balance objectives.

Reviewer comment (Theory):
After Eq (12), the reviewer is unclear on what Lemma 1 has to do with the statement that the expected value of a score is 0.

Author response:
* $E_{P_{F,\mu}(\tau;\theta_F)}[\nabla_{\theta_F}c(\tau;\theta_F)]=0$ does not imply that the expected score $E_{P_{F,\mu}(\tau;\theta_F)}[c(\tau;\theta_F)]$ is zero.  Actually, $E_{P_{F,\mu}(\tau;\theta_F)}[\nabla_{\theta_F}c(\tau;\theta_F)]=E_{P_{F,\mu}(\tau;\theta_F)}[1\cdot\nabla_{\theta_F}log P_F(\tau|s_0;\theta_F)]=\nabla_{\theta_F}E_{P_{F,\mu}(\tau;\theta_F)}[1]$ by lemma 1. Furthermore $\nabla_{\theta_F}E_{P_{F,\mu}(\tau;\theta_F)}[1]=\nabla_{\theta_F}1=0$.

Reviewer comment (Writing):
There are numerous typos and citation formatting mistakes, and another round of proofreading is recommended.

Author response:
* We thank the reviewer to point out the typos and have corrected the typos.

Reviewer comment (Novelty):
It seems to me the related work can be improved, for example, what is the most relevant standard RL algorithm? And how is your proposed policy gradient method different in design? Additionally, GFlowNet and RL are discussed together in some other papers in the literature, such as the "GFlowNet Foundations" by Bengio et al. How is the analysis in your work related to these previous works?

Author response:
### Related work:
  * We have improved the related work section to include the most relevant RL methods. Besides, we have thoroughly gone through the paper by Bengio et al. (2021b)[1]. We believe that only Section 7.2 formally talks about the relationship between GFlowNet and RL. 
    1. The first point of this section is that the traditional RL method aims at finding the highest-rewarding trajectories that render the maximization of expected return $R$ by all the probability mass on the highest-return trajectories. GFlowNets, however, try to assign proper probabilities to all possible trajectories according to their terminating reward, so that the trajectories end in terminating state $x$ with the probability proportion to $R(x)$. This fact is also emphasized on page 33, before the beginning of Section 4.7, PPO, an RL method that had previously been used for generating molecular graphs, tends to focus on a single mode of the reward function. 

     2. The second point is that Maximization Entropy (MaxEnt) RL can be understood as learning a policy such that the terminating state distribution $P^{\top}(x)\propto n(x)R(x)$, where $n(x)$ is the number of paths in the DAG of all trajectories that lead to $x$.  As explained by the authors, existing RL methods perform different tasks from GFlowNets, which learn a policy such that $P^{\top}(x)\propto R(x)$. Thus, GFlowNets (Bengio et al., 2021a)[2] originally do not fit into the traditional reinforcement learning framework as it lacks the formal definition of RL components. By contrast, our work formally bridges the "flow" in GFlowNets to RL by defining non-stationary rewards, allowing us to perform the  tasks as GFlowNets but in RL formulations.

Reviewer comment (Experiments):
I think more ablation on the proposed method with different hyperparameters can help us understand better how it can be applied/tuned to different problems.

Author response:
### Ablations:  
  * We think the ablation study is used to understand the impact of different components or features of a model on its overall performance. As we did not propose new components in policy-based optimization, we only compared our work with different GFlowNet algorithms and did not conduct an ablation study. We would truly appreciate your advice on how we may conduct the ablation study if there are specific modular components in our method requiring more careful investigation.

Reviewer comment (Presentation):
End of page 5, a figure reference seems to be broken. Is this figure in the paper?

Author response:
* We have fixed the broken citation of the figure.

Reviewer comment (Novelty):
Can you elaborate on why is RL a good strategy to use for these problems and what unique advantages it can bring? Are there concrete problems where they have to be solved with RL and not other strategies?

Author response:
* Firstly, the workflow of existing methods for GFlowNet training follows the logic of value-based RL methods. Secondly, we added Appendix B.4 to further explain that when the sampler policy $P_D=P_F$, TB-based methods correspond to estimate the $Q$ function empirically and at best, do variance reduction by a baseline $C$ which is constant w.r.t $P_F$. By contrast, $Q$ is approximated functionally, and $V$ serves as a functional baseline for variance
reduction in our policy-based method, which typically renders more robust gradient estimation (Schulman et al., 2015)[3]. Besides, the major advantage of TB-based training is allowing off-line training (i.e $P_D\neq P_F$), which is also allowable by policy-based RL methods (Degris et al., 2012)[4], for example, by importance sampling.

Reviewer comment (Novelty):
The related work section could be made more exhaustive by adding the other GFlowNet losses and their references.

Author response:
In the related work section, we have added references to all GFlowNet losses mentioned in this paper.

Reviewer comment (Experiments):
It will be useful to expand the number of environment configurations. For hypergrid, N=2 and N=3 are the only options used and using a higher value will help.

Author response:
Following Malkin et al. (2022)[1] who consider these cases with $N=2,3,4$, we conduct an additional experiment with $N=4$ and $H=32$.

Reviewer comment (Experiments):
Previous work has analyzed number of states visited for hypergrid domain, which has not been included here.

Author response:
During the training process, training data is sampled in a trajectory-wise manner for both the existing GFlowNet training methods and our RL methods. Thus, we believe training curves w.r.t the number of sampled trajectories is a better way to compare the convergence rate. For example, the DAG graphs for Hyper-grid experiments are not graded. The trajectory length varies across trajectory samples, and the number of sampled states varies in each training iteration accordingly.

Reviewer comment (Experiments):
Adding other GFlowNet based baselines, such as Detailed Balance, would be useful as it is an important and commonly used objective.

Author response:
We update the existing results by adding training results under the Detailed Balance objectives.

Reviewer comment (Experiments):
Some of the domains that were used in the previous work have been not included here. In order to corroborate that the proposed method can be used over different settings, including them would be helpful.

Author response:
We redesign the bit-sequence design experiment as bio-sequence design experiments (including nucleotide strings and molecule graphs) based on real-world datasets in  Shen et al.,2023[2].