CF LOW N ETS : C ONTINUOUS C ONTROL WITH G ENERATIVE F LOW N ETWORKS
A BSTRACT
Generative flow networks (GFlowNets), as an emerging technique, can be used as an alternative to reinforcement learning for exploratory control tasks.
GFlowNets aims to sample actions with a probability proportional to the reward, similar to sampling different candidates in an active learning fashion.
However, existing
GFlowNets cannot adapt to continuous control tasks because GFlowNets need to form a DAG and compute the flow matching loss by traversing the inflows and outflows of each node in the trajectory.
In this paper, we propose generative continuous flow networks (CFlowNets) that can be applied to continuous control tasks.
First, we present the theoretical formulation of CFlowNets.
Then, a training framework for CFlowNets is proposed, including the action selection process, the flow approximation algorithm and the continuous flow matching loss function.
Afterward, we theoretically prove the error bound of the flow approximation.
The error decreases rapidly as the number of flow samples increases.
Finally, experimental results on continuous control tasks demonstrate the performance advantages of CFlowNets compared to many reinforcement learning methods, especially regarding exploration ability
I NTRODUCTION
As an emerging technology, generative flow networks (GFlowNets) (Bengio et al., 2021a;b) can make up for the shortcomings of reinforcement learning (Kaelbling et al., 1996; Sutton & Barto, 2018) on exploratory tasks.
Specifically, based on the Bellman equation (Sutton & Barto, 2018), reinforcement learning is usually trained to maximize the expectation of future rewards; hence the learned policy is more inclined to sample action sequences with higher rewards.
In contrast, the training goal of GFlowNets is to approximately sample candidate actions with probability proportional to a given reward function, rather than generating a single high-reward action sequence (Bengio et al., 2021a).
This is more like sampling different candidates in an active learning setting (Bengio et al., 2021b), thus better suited for exploration tasks.
Unfortunately, GFlowNets currently cannot support continuous tasks.
The main reason is because
GFlowNets need to structure the state transitions of trajectories into a directed acyclic graph (DAG) structure.
Each node in the graph structure corresponds to a different state, and actions correspond to transitions between different states, that is, an edge connecting different nodes in the graph.
Obviously, the number of nodes in this graph structure is limited, and each edge can only correspond to one discrete action.
This results in GFlowNets being limited to discrete and deterministic environments.
However, in real environments, the state and action spaces are continuous for many tasks, such as quadrupedal locomotion (Kohl & Stone, 2004), autonomous driving (Kiran et al., 2021; Shalev-Shwartz et al., 2016; Pan et al., 2017), or dexterous in-hand manipulation (Andrychowicz et al., 2020).
Moreover, the reward distributions corresponding to these environments may be mul- timodal, requiring more diversity exploration.
The needs of these environments closely match the strengths of GFlowNets, so addressing the discrete spatial constraints of GFlowNets is critical.
In this paper, we propose generative continuous flow networks, named CFlowNets for short, for continuous control tasks to generate policies that can be proportional to continuous reward functions.
Applying GFlowNets to continuous control tasks is exceptionally challenging.
In generative flow networks, the transition probability is defined as the ratio of action flow and state flow.
For discrete state and action spaces, we can form a DAG and compute the state flow by traversing a node’s incoming and outgoing flows.
Conversely, it is impossible for continuous tasks to traverse all stateaction pairs and corresponding rewards.
To address this issue, we propose a method of sampling actions with probabilities roughly proportional to the output of the flow network. Then, we propose an efficient way to approximate the continuous inflows and outflows, where we use a deep neural network to predict the parent nodes of each state in the sampled trajectory. Finally, we present an approximate continuous flow matching loss function, which can efficiently train continuous flow networks.
The main contributions of this paper are summarized as the following:
Main Contributions: 1) We present the theoritical formulation of CFlowNets and propose the continuous flow matching condition theorem.
Based on this, a loss function for training CFlowNets is presented; 2) We present an efficient way to sample actions with probabilities approximately proportional to the output of the flow network, and propose a flow sampling approach to approximate continuous inflows and outflows, which allows us to construct a continuous flow matching loss; 3) We theoretically analyze the error bound between sampled flows and inflows/outflows, and the tail becomes minor as the number of flow samples increases; 4) We conduct experiments based on continuous control tasks to demonstrate that CFlowNets can outperform current state-of-the-art RL algorithms, especially in terms of exploration capabilities.
PRELIMINARIES 2. M ARKOV D ECISION P ROCESS
A stochastic, discrete-time and sequential decision task can be described as a Markov Decision
Process (MDP) , which is canonically formulated by the tuple:
[Equation]
In the process, s ∈ S represents the state space of the environment.
At each time step, agent receives a state s and selects an action a on the action space A .
This results in a transition to the next state s ′ according to the state transition function P ( s ′ | s, a ) :
S ×
A × S → [0 , 1] .
Then the agent gets the reward r based on the reward function R ( s, a ) : S ×
A → R .
A stochastic policy π maps each state to a distribution over actions π ( ·| s ) and gives the probability π ( a | s ) of choosing action a in state s .
The agent interacts with the environment by executing the policy π and obtaining the admissible trajectories { ( s t , a t , r t , s t +1 ) } nt =1 , where n is the trajectory length.
The goal of an agent is to maximize the discounted return E s 0: n ,a 0: n [ (cid:80) ∞ t =
γ t r t | s
=
s, a 0 = a, π ] , where E is the expectation over the distribution of the trajectories and γ ∈ [0 , 1) is the discount factor.
2. G ENERATIVE F LOW N ETWORK
GFlowNet sees the MDP as a flow network, that is, leverages the DAG structure of the MDP.
Define s ′ =
T ( s, a ) and F ( s ) as the node’s transition and the total flow going through s .
Define an edge/action flow F ( s, a ) =
F ( s → s ′ )
as the flow through an edge s → s ′ .
The training process of vanilla GFlowNets need to sum the flow of parents and children through nodes (states), which depends on the discrete state space and discrete action space.
The framework is optimized by the following flow consistency equations:
[Equation]
which means that for any node s , the incoming flow equals the outgoing flow, which is the total flow
F ( s ) of node s .
CF LOW N ETS : T HEORETICAL F ORMULATION
Considering a continuous task with tuple ( S , A ) , where S denotes the continuous state space and A denotes the continuous action space.
Define a trajectory τ =
( s 1 , ..., s n ) in this continuous task as a sequence sampled elements of S such that every transition a t : s t → s t +1 ∈ A .
Further, we define an acyclic trajectory τ =
( s 1 , ..., s n ) as a trajectory satisfies the acyclic constraint:
∀ s m ∈ τ, s k ∈ τ, m ̸
= k , we have s m ̸
= s k .
Denote s 0 and s f respectively as the initial state and the final state related with the continuous task ( S , A ) , we define the complete trajectory as any sampled acyclic trajectory from ( S , A ) starting in s 0 and ending in s f .
Correspondingly, a transition s → s f into the final state is defined as the terminating transition, and F ( s → s f ) is a terminating flow.
A trajectory flow F ( τ ) : τ (cid:55)→ R + is defined as any nonnegative function defined on the set of complete trajectories τ .
For each trajectory τ , the associated flow F ( τ ) contains the number of particles sharing the same path τ .
In addition, the tuple ( S , A , F ) is called a continuous flow network.
Define the parent set P ( s t ) of a state s t as the set contains all of the direct parents of s t that could make a direct transition to s t under the acyclic constraint, i.e., P ( s t ) =
{ s ∈ S : T ( s, a ∈ A ) =
s t , T ( s, a ∈
A ) / ∈ { s 0 , ..., s t − 1 }} , where T ( s, a )
= s t indicates an action a that could make a transition from state s to attain s t , and T ( s, a ∈ A ) / ∈ { s 0 , ..., s t − 1 } means that the state cannot be transferred to a preexisting state, otherwise a cyclic will occur.
Similarly, define the child set C ( s t ) of a state s t as the set contains all of the direct children of s t that could make a direct transition from s t under the acyclic constraint, i.e., C ( s t ) = { s ∈ S : T ( s t , a ∈ A )
=
s, T ( s t , a ∈ A ) / ∈ { s 0 , ..., s t }} .
Then, we have the following continuous flow definitions.
Definition 1 (Continuous State Flow) .
The continuous state flow F ( s ) : S (cid:55)→ R is the integral of the complete trajectory flows passing through the state:
[Equation]
Definition 2 (Continuous Inflows) .
For any state s t , its inflows are the integral of flows that can reach state s t under the acyclic constraint, i.e.,
[Equation]
Definition 3 (Continuous Outflows) .
For any state s t , the outflows are the integral of flows passing through state s t with all possible actions a ∈ A under the acyclic constraint, i.e.,
[Equation]
[Equation]
Based on the above definitions, we can define the transition probability P ( s → s ′ | s ) of edge s → s ′ as a special case of conditional probability introduced in Bengio et al.
(2021b).
In particular, the forward transition probability is given by
[Equation]
Similarly, the backwards transition probability is given by
[Equation]
For any trajectory sampled from a continuous task ( S , A ) , we have
[Equation]
and we further have
[Equation]
Given any trajectory τ = ( s 0 , ..., s n , s ) that starting in s 0 and ending in s , a Markovian flow (Bengio et al., 2021b) is defined as the flow that satisfies
[Equation]
and the corresponding flow network ( S , A , F ) is called a Markovian flow network (Bengio et al., 2021b).
Then, we present Theorem 1, which is proved in the appendix A.1.
Theorem 1 (Continuous Flow Matching Condition) .
Consider a non-negative function ˆ F ( s, a ) taking a state s ∈ S and an action a ∈ A as inputs.
Then we have ˆ F corresponds to a flow if
and only if the following continuous flow matching conditions are satisfied:
[Equation]
Furthermore, ˆ F uniquely defines a Markovian flow F matching ˆ F such that
[Equation]
Theorem 1 means that as long as any non-negative function satisfies the flow matching conditions, a unique flow is determined.
Therefore, for sparse reward environments, i.e., R ( s ) = 0 , ∀ s ̸
= s f , we can obtain the target flow by training a flow network that satisfies the flow matching conditions.
Such learning machines are called generative Continuous Flow Networks, named CFlowNets for short, and we have the following continuous loss function:
[Equation]
where I ( · ) denotes the indicator function.
However, obviously, the above continuous loss function cannot be directly applied in practice.
Next, we propose a method to approximate the continuous loss function based on the sampled trajectories to obtain the flow model.
CF LOW N ETS : T RAINING F RAMEWORK
For continuous tasks, it impossible to access all state-action pairs to calculate the continuous inflows and outflows.
In the following, we propose the CFlowNets training framework to address this problem, which includes an action sampling process, a flow matching approximation process.
Then, CFlowNets can be trained based on an approximate flow matching loss function.
4. O VERALL F RAMEWORK
The overview framework of CFlowNets is shown in Figure 1, including the environment interaction, flow sampling, and training procedures.
During the environment interaction phase (Left part of
Figure 1), we sample an action probability buffer based on the forward-propagation of CFlowNets, from which we can sample an action with probability proportional to the reward.
We name this process the action selection procedure, as detailed in Section 4.2.
After acquiring the action, the agent can interact with the environment to update the state, and this process repeats several steps until the complete trajectory is sampled.
Once a buffer of complete trajectories is available, we randomly sample K actions and compute the child states to approximately calculate the outflows.
For the inflows, we use these sampled actions together with the current state as the input to the deep neural network G to estimate the parent states.
Based on these, we can approximately determine the inflows.
We name this process the flow matching approximation procedure (Middle part of Figure 1), as detailed in Section 4.3.
Finally, based on the approximate inflows and outflows, we can train a CFlowNet based on the continuous flow matching loss function (Right part of Figure 1), as details in Section 4.4.
The pseudocode is provided in Appendix B.
[Figure]
4. A CTION S ELECTION P ROCEDURE
Starting from an empty set, CFlowNets aim to obtain complete trajectories τ
= ( s 0 , s 1 , ..., s f ) ∈ T by iteratively sampling a t ∼ π ( a t | s t ) =
F ( s t ,a t ) F ( s t ) with tuple { ( s t , a t , r t , s t +1 ) } ft =0 .
However, it is difficult to sample trajectories strictly according to the corresponding probability of a t , since the actions are continuous, we cannot get the exact action probability distribution function based on the flow network F ( s t , a t ) .
To solve this problem, at each state s t , we first uniformly sample M actions from A and generate an action probability buffer P = { F ( s t , a i ) } M i =1 , which is used as an approximation of action probability distributions.
Then we sample an action from P according to the corresponding probabilities of all actions.
Obviously, actions with larger F ( s t , a i ) will be sampled with higher probability.
In this way, we approximately sample actions from a continuous distribution according to their corresponding probabilities.
Remark 1.
After the training process, for tasks that require a larger reward, we can sample actions with the maximum flow output in P during the test process to obtain a relatively higher reward.
How the output of the flow model is used is flexible, and we can adjust it for different tasks.
4. F LOW M ATCHING A PPROXIMATION
Once a batch of trajectories B is available, to satisfy flow conditions, we require that for any node s t , the inflows (cid:82) s,a : T ( s,a )
=
s t F ( s, a )d s d a equals the outflows (cid:82) a ∈A F ( s t , a )d a , which is the total flow F ( s t ) of node s t .
However, obviously, we cannot directly calculate the continuous inflows and outflows to complete the flow matching condition.
An intuitive idea is to discretize the inflows and outflows based on a reasonable approximation and match the discretized flows.
To do this, we sample K actions independently and uniformly from the continuous action space A and calculate corresponding F ( s t , a k ) , k
= 1 , ..., K as the outflows, i.e., we use the following approximation:
[Equation]
where µ ( A ) denotes the measure of the continuous action space A .
By contrast, an approximation of inflow is more difficult since we should find the parent state first.
To solve this problem, we construct a transaction deep neural network G parameterized by ϕ with ( s t +1 , a t ) as the input while s t as the output, and train this network based on B with the MSE loss.
The network G is usually easy to train, and we can obtain a high-precision network G through simple pre-training.
As the training progresses, we can also occasionally update G based on the sampled trajectories to ensure accuracy.
Then, the inflows can be calculated approximately:
[Equation]
Remark 2.
Recall that in Definitions 2 and 3, we define acyclic constraints that need to be satisfied when selecting possible actions and finding parents.
But this constraint is only for the convenience of theoretical analysis.
Here we ignore this constraint in equation 14 and equation 15 because the probability of picking continuous actions to form a cyclic is almost zero.
This is understandable since ∀ t, µ ( { s 0 , ..., s t } ) = 0 and µ ( A ) = µ ( A\{ s 0 , ..., s t } ) . Next, by assuming that the flow function F ( s, a ) is Lipschitz continuous in Assumption 1, we could provide a non-asymptotic analysis for the error between the sample inflows/outflows and the true inflows/outflows.
Theorem 2 establishes the error bound between the sample outflows (resp.
inflows) and the actual outflows (resp.
inflows) in the tail form and shows that the tail is decreasing exponentially.
Furthermore, the tail gets much smaller with the increase of K , which means the sample outflows (resp.
inflows) are a good estimation of the actual outflows (resp. inflows).
Assumption 1.
Assume the function F ( s, a ) is Lipschitz continuous, i.e.,
[Equation]
16) where L s is a constant related to s .
Theorem 2.
Let { a k } Kk =1 be sampled independently and uniformly from the continuous action space
A .
Assume G ϕ ⋆ can optimally output the actual state s t with ( s t +1 , a t ) .
For any bounded continuous action A and any state s t ∈ S , we have
[Equation]
and
[Equation]
where L s t is the Lipschitz constant of the function F ( s t , a ) and diam( A ) denotes the diameter of the action space A .
4. L OSS F UNCTION
Based on equation 14 and equation 15, the continuous loss function can be approximated by
[Equation]
where θ is the parameter of the flow network F ( · ) and λ =
K/µ ( A ) .
Note that the action set A changes slightly with the transition of nodes in the trajectory, because the action needs to satisfy the acyclic constraint T ( s t , a ) ∈ { s 0 , ..., s t } . However, the measure of A is a constant because ∀ t, µ ( A ) = µ ( A\{ s 0 , ..., s t } ) .
This shows that as long as the number of samples K is large enough, the discrete flow matching loss is equivalent to adding a fixed value shaping operation to the reward of the continuous flow matching loss, i.e., λ is more of a hyperparameter here. It doesn’t need to be exactly equal to K/µ ( A ) but more like adding a fixed value shaping to the reward.
It is noteworthy that the magnitude of the state flow at different locations in the trajectory may not match.
For example, the initial node flow is likely to be larger than the ending node flow.
To solve this problem, inspired the log-scale loss introduced in GFlowNets (Bengio et al., 2021a), we can modified equation 19 into:
[Equation]
where ϵ is a hyper-parameter that helps to trade off small versus large flows and helps avoid the numerical problem of taking the logarithm of tiny flows.
R ELATED W ORKS
Generative Flow Networks.
Generative flow networks are proposed to enhance exploration capabilities by generating policies that sample objects through discrete action sequences with probabilities proportional to a predefined reward function (Bengio et al., 2021b;a).
Since the network only samples actions based on the distribution of the corresponding rewards, rather than focusing only on actions that maximize rewards such as reinforcement learning, it can perform well on tasks with more diverse reward distributions, and has been successfully applied to molecule generation (Bengio et al., 2021a; Malkin et al., 2022; Jain et al., 2022), discrete probabilistic modeling (Zhang et al., 2022b) and structure learning (Deleu et al., 2022). In Malkin et al.
(2022), the trajectory balance loss is proposed for GFlowNets to explore the capabilities of previously used objectives.
The connection between deep generative models and GFlowNets is discussed in Zhang et al.
(2022a) through the lens of Markov trajectory learning.
In Bengio et al.
(2021b), an idea based on hybrid state is presented to make GFlowNets suitable for continuous tasks.
This idea is mainly based on decomposing the continuous state space into a discrete state space plus continuous residuals.
However, the decomposition process often requires gridding processing, which is not suitable for continuous spaces at high latitudes since the state and action dimensions would explode exponentially.
Continuous Reinforcement Learning.
Policy gradient algorithms are widely used for reinforcement learning problems with continuous action spaces.
The deterministic policy gradient (DPG) (Silver et al., 2014) algorithm is an actor-critic (Grondman et al., 2012;
Rosenstein et al., 2004) method that uses an estimate of the learned value Q ( s, a ) to train a deterministic policy µ : S →
A parameterized by θ µ .
Compared with CFlowNets, the policy is updated by applying the chain rule to the expected return J from the start distribution with respect to the policy parameters: [Equation]
[Equation]
where D is the replay buffer.
The policy aims to maximize the expectation of future rewards, which are estimated by Q -learning.
In this setting, the trajectories generated by the policy may be relatively homogeneous.
However, for CFlowNets, by harnessing the power of the flow network, the probability of an action being sampled is proportional to the corresponding reward, resulting in more diverse trajectories that are beneficial for exploring the environment.
Later, deep DPG (DDPG) (Lillicrap et al., 2015) improves DPG and has good sample-efficient property but suffers from extreme brittleness and hyperparameter sensitivity.
Therefore, it is difficult to extend DDPG to complex, high-dimensional tasks.
To improve DDPG, twin delayed DDPG (TD3) (Fujimoto et al., 2018) adopts an actor-critic framework and considers the interaction between value update and function approximation error and in the policy.
There are also some policy gradient (Sutton et al., 1999; Kohl & Stone, 2004; Khadka & Tumer, 2018) based algorithms that can be adapted for continuous tasks, such as proximal policy optimization (PPO)
(Schulman et al., 2017) algorithms, asynchronous advantage actor-critic (A3C) (Stooke & Abbeel, 2018), and importance weighted actor-learner architecture (IMPALA) (Espeholt et al., 2018).
PPO has the benefits of trust region policy optimization (Schulman et al., 2015), enabling multiple batches of data to be updated together.
Therefore, it is simpler to implement, more general, and has lower sample complexity.
Recently, phasic policy gradient (PPG) (Cobbe et al., 2021) is proposed to decouple the training between policy and value function while keeping their feature sharing, and PPG optimizes each objective with an appropriate level of sample reuse to improve sample efficiency.
All of these improved policy gradient methods can be classified as aiming at maximizing reward, so none of them are better suited for exploration tasks than CFlowNets.
Furthermore, some maximum entropy (Pitis et al., 2020; Haarnoja et al., 2018a; Hazan et al., 2019;
Yarats et al., 2021) based reinforcement learning algorithms can also be adapted for continuous tasks, such as soft actor-critic (SAC) (Haarnoja et al., 2018b).
By maximizing the expected reward and entropy, the actor network of SAC can successfully complete tasks while acting as randomly as possible.
The difference between CFlowNets and SAC is: 1) Although the goal of SAC is to make policy proportional to return based on maximum entropy, the actor network of SAC outputs the mean and variance of the learned policy.
Therefore, what SAC learns is not the true distribution of strictly proportional returns; 2)
The return proportional to the policy in SAC can be understood as an accumulation of multiple samplings of the reward. This is different from being directly proportional to reward. In Bengio et al.
(2021a), the difference between the two is discussed in Proposition 4, taking the tree structure as an example.
E XPERIMENTS
We investigate the performance of our CFlowNets by conducting experiments on several continuous control tasks with sparse rewards.
The experimental setup is first introduced.
Then we compare CFlowNets with a few state-of-the-art baseline RL algorithms, such as DDPG (Lillicrap et al., 2015), TD3 (Fujimoto et al., 2018), PPO (Schulman et al., 2017), and SAC (Haarnoja et al., 2018b).
6. E XPERIMENTAL S ETUP
We conduct our experiments on three continuous control tasks: Point-Robot-Sparse, Reacher-GoalSparse, and Swimmer-Sparse.
Point-Robot-Sparse is a continuous navigation task.
FigureThe visualization of these environments is shown in Figure 4. In this task, the agent starts at the starting coordinate (0 , 0) and moves towards the target coordinate one step at a time. This experiment has two target coordinates (5 , 10) and (10 , 5) . The environment has a maximum episode length of 12, and the environment returns a reward only when the last step is reached.
Rewards are issued by measuring the distance between the agent’s current position and the target node, and the closer the distance, the greater the reward.
Each time the agent can choose to take a step from any angle to the upper right.
Both Reacher-Goal-Sparse and Swimmer-Sparse are adapted from OpenAI Gym’s MuJoCo environment.
In the Reacher-Goal-Sparse environment, “Reacher” is a two-jointed robotic arm.
The goal is to move the robot’s end effector (called fingertip) close to a randomly generated target.
The ”swimmer” is suspended in a two-dimensional pool, and the goal is to move as fast as possible towards the right or left by taking the action that applies torque between links of the rotors.
We set the maximum number of steps to 50 for these two environments.
For Reacher-Goal-Sparse, when the last step is reached, the environment returns a reward that measures how far the agent is from the randomly generated target.
The closer the agent is to the target, the greater the reward.
For Swimmer-Sparse, the farther to the left or right from the starting point, the greater the reward returned.
More details of parameter setting are provide in Appendix C.
6. E XPERIMENTAL P ERFORMANCE
[Figure]
E s t i m a t ed
R e w a r d V a l ue
True Reward
DDPGTD3SACCFlowNets
Figure 2 illustrates the distributions of learned policies for
CFlowNets and RL algorithms.
All curves are max-min normalized.
The gray curve is the ground truth of reward distribution generated by the agent’s different actions when it goes to coordinates (7 , 7) , which indicates that the optimal action here is to go right or up.
The red curve shows the flow network output of CFlowNets under different actions, indicating that CFlowNets have an excellent fitting ability to the reward.
In contrast, other reinforcement learning algorithms have difficulty fitting the actual reward distribution well.
Figures 3(a)-(c) show the number of valid-distinctive trajectories explored as training progresses in Point-Robot-Sparse,
Reacher-Goal-Sparse, and Swimmer-Sparse environment, respectively.
After a certain number of training epochs, 10000 trajectories are collected.
A valid-distinctive trajectory is defined as a reward above a threshold δ r while the MSE between the trajectory and other trajectories is greater than another threshold δ mse .
That is, if the returns of both trajectories are high, but the two are close and the MSE is small, we consider it only one valid-distinctive exploration.
δ r in PointRobot-Sparse, Reacher-Goal-Sparse, and Swimmer-Sparse is set as 0.5, -0.2, 5.0, respectively.
δ mse in Point-Robot-Sparse, Reacher-Goal-Sparse, and Swimmer-Sparse is set as 0.02, 4.0, 1.0, respectively.
As can be seen from the figure, DDPG, TD3 and PPO have the worst exploration ability, only one valid-distinctive trajectory is generated.
SAC explores better at the beginning of training, and decreases as the training progresses and gradually converges.
In contrast, the exploration ability of
CFlowNets is very outstanding, the number of trajectories explored far exceeds other algorithms, and the exploration ability has been stable as the training progresses.
Figures 3(d)-(f) indicate the rewards during the training process in Point-Robot-Sparse, ReacherGoal-Sparse, and Swimmer-Sparse environment, respectively.
The shaded region represents 95% confidence interval across 5 runs.
Figure 3(d) and Figure 3(e) show that CFlowNets has the fastest and more stable upward trend, and the final reward is ahead of that of other algorithms by a large margin.
In contrast, CFlowNets do not perform as well as other algorithms in Figure 3(f).
Since the rewards in Point-Robot-Sparse and Reacher-Goal-Sparse are more evenly distributed, so these two tasks are more inclined to exploration.
CFlowNets has better exploration ability and hence can converge stably.
As for Swimmer-Sparse, its reward distribution is relatively steep, and sampling near the maximum reward can achieve faster convergence.
It is reasonable for CFlowNets to perform worse than RL on this task in terms of reward.
However, in this environment, CFN can still maintain a good exploration ability.
C ONCLUSION
In this paper, we propose generative continuous flow networks to enhance exploration in continuous control tasks.
The theoretical formulation of CFlowNets is first presented.
Then, a training framework for CFlowNets is proposed, including the action selection process, the flow approximation algorithm, and the continuous flow matching loss function.
Theoretical analysis shows that the error of the flow approximation decreases rapidly as the number of flow samples increases.
Experimental results on continuous control tasks illustrate the performance advantages of CFlowNets compared to many reinforcement learning methods.
Especially in the exploration ability, the effect of CFlowNets far exceeds other state-of-the-art reinforcement learning algorithms.
Limitations: Similar to GFlowNets, CFlowNets aims to sample actions with probabilities proportional to the reward distribution, rather than selecting actions with maximizing rewards.
Therefore, CFlowNets are more suitable for exploration-biased tasks.
It does not perform as well as reinforcement learning on tasks that aim to maximize reward.
Of course, the purpose of CFlowNets is not to completely replace reinforcement learning, but as a supplement to reinforcement learning, giving a new option for continuous control tasks.
[Figure]
A P ROOFS A. P ROOF OF T HEOREM
Theorem 1.
(Continuous Flow Matching Condition).
Consider a non-negative function ˆ F ( s, a ) taking a state s ∈ S and an action a ∈ A as inputs.
Then we have ˆ F corresponds to a flow if
and only if the following continuous flow matching conditions are satisfied:
[Equation]
Furthermore, ˆ F uniquely defines a Markovian flow F matching ˆ F such that
[Equation]
Proof.
The proof is an extension of that of Proposition 19 in Bengio et al.
(2021b)
to the continuous case.
We first prove the necessity part of the proof.
Given a flow network, for non-initial and nonfinal nodes on a trajectory, the set of complete trajectories passing through state s ′ is the union of the sets of trajectories going through s → s ′ for all s ∈ P ( s ′ ) , and also is the union of the sets of trajectories going through s ′
→ s ′′ for all s ′′ ∈ C ( s ′ ) , i.e.,
[Equation]
Then we have
[Equation]
and
[Equation]
Then we finish the necessity part.
Next we show sufficiency.
Let ˆ Z =
ˆ F ( s 0 ) be the partition function and ˆ P F be the forward probability function, then there exists a unique Markovian flow
F with forward transition probability function P F = ˆ P F and partition function Z according to
Proposition 18 in Bengio et al.
(2021b) [Equation] and such that
[Equation]
where s n +1 =
s f .
In addition, according to Lemma
[Equation]
we have
[Equation]
Lemma 1.
Considering a continuous task ( S , A ) , where we have the transition probabilities defined in equation 5 and equation 6.
Define T s,f and T 0 ,s as the set of trajectories sampled from a continuous task starting in s and ending in s f ; and starting in s 0 and ending in s , respectively.
Then we have
[Equation]
[Equation]
[Equation]
Thus, we have for s ′ ̸
= s 0 :
[Equation]
Combine equation 27 with P F = ˆ P F yields ∀ s → s ′ ∈ A , F ( s → s ′ ) .
Finally, according to
Proposition 16 in Bengio et al.
(2021b)
, for any Markovian flow F ′ matching ˆ F on states and edges, we have F ′ ( τ ) =
F ( τ ) , which shows the uniqueness property.
Then we complete the proof.
A. P ROOF OF T HEOREM
Theorem 2.
Let { a k } Kk =1 be sampled independently and uniformly from the continuous action space
A .
Assume G ϕ ⋆ can optimally output the actual state s t with ( s t +1 , a t ) .
For any bounded continuous action A and any state s t ∈ S , we have
[Equation]
and
[Equation]
where L s t is the Lipschitz constant of the function F ( s t , a ) and diam( A ) denotes the diameter of the action space A .
Proof.
First, we show that the expectation of sample outflow is the true outflow and the expectation of sample inflow is the true inflow in Lemma 2.
Lemma 2.
Let { a k } Kk =1 be sampled independently and uniformly from the continuous action space
A .
Assume G ϕ ⋆ can optimally output the actual state s t with ( s t +1 , a t ) .
Then for any state s t ∈ S , we have
[Equation]
[Equation]
and
E
[Equation]
Then, define the following terms:
[Equation]
and
[Equation]
Note that the variables { Γ k } Kk =1 are independent and E [Γ k ] = 0 , k = 1 , . . .
, K according to Lemma 2.
So the following equations hold
[Equation]
and
[Equation]
Since F ( s, a ) is a Lipschitz function, we have
[Equation]
Together with the fact that for any pair of (s,a) satisfying T ( s, a ) = s t , s is unique if we fix a , we have
[Equation]
Lemma 3 (Hoeffding’s inequality, Vershynin (2018)) .
Let x 1 , . . .
, x K be independent random variables.
Assume the variables { x k } Kk =1 are bounded in the interval [ T l , T r ] .
Then for any t > 0 ,we have
[Equation]
Incorporating T r =
L st µ ( A )
[Equation]
K and T r = − L st µ ( A )diam( [Equation] K in Lemma 3 completes the proof.
A. P ROOF OF L EMMA
Lemma 1.
Considering a continuous task ( S , A ) , where we have the transition probabilities defined in equation 5 and equation 6.
Define T s,f and T 0 ,s as the set of trajectories sampled from a continuous task starting in s and ending in s f ; and starting in s 0 and ending in s , respectively.
Then we have
[Equation]
Proof.
We show by strong induction that equation 41 holds, mainly following the proof of Lemmain Bengio et al.
(2021b), and then extending to equation 42 is trivial.
Define d as the maximum trajectory length in T s,f , s ̸ = s f , we have:
Base cases: If d = 1 , then
[Equation]
holds by noting T s,f = { ( s → s f ) } .
Induction steps: Consider d > 1 , by noting equation {+
+} we have
[Equation]
where the last equality follows by the induction hypotheses.
A. P ROOF OF L EMMA
Lemma 2.
Let { a k } Kk =1 be sampled independently and uniformly from the continuous action space
A .
Assume G ϕ ⋆ can optimally output the actual state s t with ( s t +1 , a t ) .
Then for any state s t ∈ S , we have E
[Equation]
[Equation]
and
E
[Equation]
Proof.
Since { a k } Kk =1 is sampled independently and uniformly from the continuous action space A , then we have [Equation]
[Equation]
Therefore, we obtain
[Equation]
Note that for any pair of (s,a) satisfying T ( s, a )
= s t , s is unique if we fix a . So we have
[Equation]
Therefore, we get
[Equation]
B P SEUDOCODE OF CF LOW N ETS
For clarity, we show pseudocode for CFlowNets in Algorithm 1.
Initialize :
Flow network θ ; a pretrained transaction network ϕ ; and empty buffer D and P 1: repeat 2:
Set t = 0 , s = s3: while s ̸ = terminal and t<T do 4:
Uniformly sample M actions { a i } Mi =1 from action space
A 5:
Compute edge flow F θ ( s t , a i ) for each a i ∈ { a i } Mi =1 to generate P 6:
Sample a t ∼ P and execute a t in the environment to obtain r t +1 and s t + 7: t = t +8: end while 9:
Store episodes { ( s t , a t , r t , s t +1 ) }
Tt =1 in replay buffer D 10:
[Optional] Fine-tuning transaction network ϕ based on D 11:
Sample a random minibatch B of episodes from D 12:
Uniformly sample K actions { a k } Kk =1 from action space
A for each state in B 13:
Compute parent states according to { G ϕ ( s, a k ) } Kk =1 for each state in B 14:
Inflows: 15:
[Equation]
Outflows or reward: 17:
[Equation]
Update flow network F θ according to equation19: until convergence
[Figure]
C E XPERIMENTAL SETUP C. VISUALIZATION OF ENVIRONMENT
As shown in Figure 4, we provide the visualization of Point-Robot-Sparse, Reacher-Goal-Sparse, and Swimmer-Sparse tasks.
C. B ASELINES
We compare our proposed CFlowNets to the following baselines:
• Deep deterministic policy gradient (DDPG) (Lillicrap et al., 2015).
https://github.
com/sfujim/TD3/blob/master/DDPG.py • Twin delayed deep deterministic policy gradient (TD3) (Fujimoto et al., 2018).
https: //github.com/sfujim/TD • Soft actor-critic (SAC) (Haarnoja et al., 2018b).
https://github.com/ denisyarats/pytorch_sac/ • Proximal policy optimization (PPO) (Schulman et al., 2017).
https://github.com/ DLR-RM/stable-baselines3/blob/master/stable_baselines3/ppo/ ppo.py
C. H YPER -P ARAMETER
We provide the hyper-parameter of all compared methods under different environments in Table 1,
Table 2, Table 3, Table
[Table]
and Table 5.
[Table]
[Table] [Table]
[Table] [Table]