PFPN: Continuous Control of Physically Simulated Characters using Particle Filtering Policy Network
Abstract
Data-driven methods for physics-based character control using reinforcement learning have been successfully applied to generate high-quality motions.
However, existing approaches typically rely on Gaussian distributions to represent the action policy, which can prematurely commit to suboptimal actions when solving highdimensional continuous control problems for highly-articulated characters.
In this paper, to improve the learning performance of physics-based character controllers, we propose a framework that considers a particle-based action policy as a substitute for Gaussian policies.
We exploit particle ﬁltering to dynamically explore and discretize the action space, and track the posterior policy represented as a mixture distribution.
The resulting policy can replace the unimodal Gaussian policy which has been the staple for character control problems, without changing the underlying model architecture of the reinforcement learning algorithm used to perform policy optimization.
We demonstrate the applicability of our approach on various motion capture imitation tasks.
Baselines using our particle-based policies achieve better imitation performance and speed of convergence as compared to corresponding implementations using Gaussians, and are more robust to external perturbations during character control.
Introduction
In the last few years, impressive results have been obtained for physics-based character control using data-driven methods under the framework of deep reinforcement learning (DRL)
[Peng et al., 2018,
Merel et al., 2017, Chentanez et al., 2018, Won et al., 2020, Bergamin et al., 2019, Ma et al., 2021,
Yin et al., 2021].
Works from [Peng and van de Panne, 2017, Reda et al., 2020, Lee et al., 2019,
Jiang et al., 2019] studied the learning and control performance with different action spaces using joint or muscle actuation formulations, and show the impact that action parameterization has on the trained policies.
Those state-of-the-art approaches for physics-based character control typically perform exploration in the action space of joint angles and control the character using PD servos to generate high-ﬁdelity motions through imitation learning.
Despite recent success, though, generating high-quality and robust animation for highly articulated characters is still a challenging task.
Due to the inﬁnite feasible action choices, controlling many degrees of is inherently ambiguous with respect to most behaviors, resulting in control problems that are under speciﬁed and highly dimensional.
Many methods have been proposed that can improve the performance of DRL for continuous control problems, e.g. distributed training [Mnih et al., 2016, Espeholt et al., 2018], hierarchical learning [Daniel et al., 2012, Haarnoja et al., 2018a], and maximum entropy regularization [Haarnoja et al., 2017, Liu et al., 2017, Haarnoja et al., 2018b].
Most of such works, though, focus on learning mechanisms to boost performance beyond the basic distribution that deﬁnes the action policy, where a Gaussian-based policy or that with a squashing function is the most common choice as the basic policy to deal with continuous action spaces.
Nevertheless, the unimodal form of a Gaussian distribution could prematurely commit to suboptimal actions when optimizing a reward function that consists of multiple competing terms and has a multimodal landscape for physics-based character control [Hamalainen et al., 2020].
Consider, for example, an articulated character that needs to learn a mapping from states to actions based on all individual joints while tracking a reference motion.
This is a very challenging task as for a given DOF and a continuous action space, we need to ﬁnd a state-dependent mean and standard deviation.
This process can be imagined as sliding the Gaussian to determine where to place it on an inﬁnite line while also shrinking and swelling the distribution.
And of course the problem becomes even more complicated as we need to do the same thing for all
DOFs and determine how they work in synchrony for a given state.
To improve the policy expressivity beyond the unimodal Gaussian, recent works in character animation use primitive actions [Peng et al., 2019], coactivations [Ranganath et al., 2019], or a mixture of experts [Won et al., 2020] , though the underlying distribution is still Gaussian.
To address the unimodality issue of Gaussian policies, in the ﬁeld of DRL, people have been exploring more expressive distributions than Gaussians, with a simple solution being to discretize the action space and use categorical distributions as multi-modal action policies [Andrychowicz et al., 2020, Ja ´ skowski et al., 2018, Tang and Agrawal, 2019].
However, as we show, such a solution cannot scale well when training control policies for physically simulated characters.
The reason is that the performance of the action-space discretization depends a lot on the choice of discrete atomic actions, which are usually picked uniformly due to lack of prior knowledge.
A simple, ﬁxed action discretization scheme typically can only provide suboptimal solutions that are unable to meet the ﬁne control demands.
In this paper, we introduce an expressive, multimodal action policy for DRL-based learning of physics-based controllers for highly articulated characters.
Following prior work, we employ a joint-actuation space and learn a mapping from states to joint target angles that are given as input to
PD servos for computing the corresponding torque values.
Instead of representing each action as a state-depending Gaussian, though, we exploit a particle approach to dynamically sample the action space during training and track the policy represented as a mixture of Gaussian distributions with state-independent components.
We refer to the resulting policy network as Particle Filtering Policy
Network (PFPN).
We evaluate PFPN on benchmarks from the DeepMimic framework [Peng et al., 2018, 2020] involving motor control tasks for a humanoid and a dog character (see Figure for some results).
Our experiments show that baselines using PFPN exhibit better imitation performance and/or speed of convergence as compared to state-of-the-art Gaussian baselines, and lead to more robust character control under external perturbation.
In addition, PFPN-trained controllers can generate motions with high visual quality for articulated characters performing motor tasks with the grace and naturalness of complex beings.
Overall, PFPN offers a great alternative to Gaussian action policies leading to state-of-the-art controllers for physically simulated characters without introducing any notable computational overhead.
Background
We consider a standard reinforcement learning setup where given a time horizon H and the trajectory τ = ( s 1 , a 1 , · · · , s H , a H ) obtained by a transition model M ( s t +1 | s t , a t ) and a parameterized action policy π θ ( a t | s t ) , with s t and a t denoting the state and action taken at time step t , respectively, the goal is to ﬁnd the policy parameters θ that maximize the cumulative reward:
[Equation]
Here, p θ ( τ ) denotes the state-action visitation distribution for the trajectory τ induced by the transient model M and the action policy π θ , and r ( τ ) = (cid:80) t r ( s t , a t ) where r ( s t , a t ) is the reward received at time step t .
We can maximize J ( θ ) by adjusting the policy parameters θ through the gradient ascent method, where the gradient of the expected reward can be determined according to the policy gradient theorem [Sutton et al., 2000], i.e.
[Equation]
where A t denotes an estimate to the reward term r t ( τ ) .
In DRL, the estimator of A t often relies on a separate network (critic) that is updated in tandem with the policy network (actor).
This gives rise to a family of policy gradient algorithms known as actor-critic.
Given a multi-dimensional continuous action space, the most common choice in current DRL baselines is to model the policy π θ as a multivariate Gaussian distribution with independent components for each action dimension.
For simplicity, let us consider a simple case with a single action dimension and deﬁne the action policy as π θ ( ·| s t ) :
= N ( µ θ ( s t ) , σ 2 θ ( s t )) .
Then, we can obtain log π θ ( a t | s t ) ∝ − ( a t − µ θ ( s t )) 2 .
Given a sampled action a t and the estimate of cumulative rewards
A t , the optimization process based on the above expression can be imagined as that of shifting µ θ ( s t ) towards the direction of a t if A t is higher than the expectation, or to the opposite direction if A t is smaller.
Such an approach, though, can easily converge to a suboptimal solution, if, for example, the reward landscape has a basis between the current location of µ θ ( s t ) and the optimal solution, or hard to be optimized if the reward landscape is symmetric around µ θ ( s t ) .
These issues arise due to the fact that the Gaussian distribution is inherently unimodal, while the reward landscape could be multi-modal [Haarnoja et al., 2017].
Indeed, Hamalainen et al.
[2020] showed that the reward landscape for high-dimensional character control tasks typically has a complex shape and is often multimodal.
This means that Gaussian policies may face difﬁculties during optimization for character control.
Particle Filtering Policy Network
In this section, we describe our Particle Filtering Policy Network (PFPN) that addresses the unimodality issues from which typical Gaussian-based policy networks suffer.
Our approach represents the action policy as a mixture distribution obtained by adaptively discretizing the action space using state-independent particles.
The policy network, instead of directly generating actions, it is tasked with choosing particles, while the ﬁnal actions are obtained by sampling from the selected particles.
3.1 Particle-Based Action Policy
We deﬁne P := {(cid:104) µ i,k , w i,k
( s t | θ ) (cid:105)| i = 1 , . . . , n ; k = 1 , . . .
, m } as a weighted set of particles for continuous control problems having an m -dimensional action space and n particles distributed on each action action space.
Here, µ i,k represents an atomic action location on the k -th dimension of the action space, and w i,k
( s t | θ ) denotes the associated weight generated by the policy network with parameters θ given the input state s t .
Let p i,k
( a i,k | µ i,k , ξ i,k ) denote the probability density function of the distribution deﬁned by the location µ i,k and a stochastic process ξ i,k for sampling.
Given P , we deﬁne the action policy as factorized across the action dimensions:
[Equation]
where a t = { a t, 1 , · · · , a t,m } , a t,k is the sampled action at the time step t for the action dimension k , and w i,k ( ·| θ ) is obtained by applying a softmax operation to the output of the policy network for the k -th dimension and satisﬁes (cid:80) i w i,k = 1 .
The state-independent parameter set, { µ i,k } , gives us an adaptive discretization scheme that can be optimized during training.
The noise parameter, ξ i,k , provides a way to generate stochastic actions during policy training.
In our implementation, we choose Gaussian distributions with a standard deviation of ξ i,k for action sampling during training and then each particle can be regarded as a state-independent Gaussian of N ( µ i,k , ξ 2 i,k ) .
Without loss of generality, we deﬁne the parameters of a particle as φ i,k = [ µ i,k , ξ i,k ] for the following discussion.
While the softmax operation gives us a categorical distribution deﬁned by w · ,k ( s t | θ ) , the nature of the policy for each dimension is a mixture distribution with state-independent components deﬁned by φ i,k .
The number of output neurons in PFPN increases linearly as the number of action dimensions increases, and thus makes it suitable for high-dimensional control problems.
Drawing samples from the mixture distribution can be done in two steps.
ﬁrst, based on the weights w · ,k ( s t | θ ) , we perform sampling on the categorical distribution to choose a particle j k for each dimension k , i.e.
[Equation]
Then, we can draw actions from the components represented by the chosen particles with noise as
[Equation]
3.2 Training
The proposed particle-based policy distribution is general and can be applied directly to any algorithm using the policy gradient method with Equation 2.
To initialize the training, due to lack of prior knowledge, the particles can be distributed uniformly along the action dimensions with a standard deviation covering the gap between two successive particles.
With no loss of generality, let us consider below only one action dimension and drop the subscript k .
Then, at every training step, each particle i will move along its action dimension and be updated by
[Equation]
where a t ∼ π P θ ( ·| s t ) is the action chosen during sampling, and c t =
A t / (cid:80) j w j ( s t | θ ) p j ( a t | φ j ) is a coefﬁcient shared by all particles on the same action dimension.
Our approach focuses only on the action policy representation in general policy gradient methods.
The estimation of A t can be chosen as required by the underlying policy gradient method, e.g. the generalized advantage estimator [Schulman et al., 2015] in PPO/DPPO and the V-trace based temporal difference error in
IMPALA [Espeholt et al., 2018].
Similarly, for the update of the policy neural network, we have
[Equation]
From the above equations, although sampling is performed on only one particle for each given dimension, all of that dimension’s particles will be updated during each training iteration to move towards or away from the location of a t according to A t .
The amount of the update, however, is regulated by the state-dependent weight w i ( s t | θ ) : particles that have small probabilities to be chosen for a given state s t will be considered as uninteresting and be updated with a smaller step size or not be updated at all.
On the other hand, the update of weights is limited by the distance between a particle and the sampled action: particles too far away from the sampled action would be considered as insigniﬁcant to merit any weight gain or loss.
In summary, particles can converge to different optimal locations near them during training and be distributed multimodally according to the reward landscape deﬁned by A t , rather than collapsing to a unimodal, Gaussian-like distribution.
3.3 Resampling
Similar to traditional particle ﬁltering approaches, our approach would encounter the problem of degeneracy [Kong et al., 1994].
During training, a particle placed near a location at which sampling gives a low A t value would achieve a weight decrease.
Once the associated weight reaches near zero, the particle will not be updated anymore (cf.
Equation 6) and become ‘dead’.
We adapt the idea of resampling from the particle ﬁltering literature [Doucet et al., 2001] to perform resampling for dead particles and reactivate them by duplicating alive target particles.
A particle is considered dead if its maximum weight over all possible states is too small,
i.e.
[Equation]
where (cid:15) is a small positive threshold number.
In practice, we cannot check w i ( s t | θ ) for all possible states, but can keep tracking it during sampling based on the observed states collected in the last batch of environment steps.
During resampling, a target particle τ i is drawn for each dead particle i independently.
We, by default, uses a weighted resampling strategy that draws a target
τ
i from the categorical distribution obtained from the average weight of each particle over the observed states, i.e.
[Equation]
Such a heuristic resampling strategy considers the importance of target particle candidates leading to a more stable learning performance, as we will show empirically by the ablation study in Appendix G.
Theorem.
Let D τ be a set of dead particles sharing the same target particle τ .
Let also the logits for the weight of each particle k be generated by a fully-connected layer with parameters ω k for the weight and b k for the bias.
The policy π P θ ( a t | s t ) is guaranteed to remain unchanged after resampling via duplicating φ i ← φ τ , ∀ i ∈
D τ , if the weight and bias used to generate the unnormalized logits of the target particle are shared with those of the dead one as follows:
[Equation]
Proof.
See Appendix B for the inference.
The theorem guarantees the correctness of our resampling process as it keep the action policy π P θ ( a t | s t ) identical before and after resampling.
If, however, two particles are exactly the same after resampling, they will always be updated together at the same pace during training and lose diversity.
To address this issue, we add some regularization noise to the mean value when performing resampling, i.e. µ i ← µ τ + ε i , where ε i is a small random number to prevent µ i from being too close to its target µ τ .
We refer to Appendix A for the pseudocode of our proposed PFPN approach.
3.4 Action-Value Based Optimization
PFPN can be directly applied on general policy gradient algorithms, like PPO [Schulman et al., 2017] and A3C [Mnih et al., 2016].
However, we note that a number of algorithms, such as DDPG [Lillicrap et al., 2015], SAC [Haarnoja et al., 2018b,c] and their variants [Haarnoja et al., 2017, Fujimoto et al., 2018], perform optimization by maximizing a state-action value Q ( s t , a t ) .
In this case, the action policy is required to be reparameterizable such that the sampled action a t can be rewritten as a function differentiable to the policy network parameter θ , and the optimization can be done through the gradient ∇ a t Q ( s t , a t ) ∇ θ a t .
Our two-step sampling method described in Section 3.1 is non-reparameterizable, because of the standard way of sampling from the categorical distribution through which Gaussians are mixed.
To address this issue and enable PFPN applicable in state-action value based off-policy algorithms, we consider the concrete distribution [Jang et al., 2016, Maddison et al., 2016] that generates a reparameterized continuous approximation to a categorical distribution.
We refer to Appendix C for the reprarameterization trick and further details on the application of PFPN to off-policy algorithms.
Experiments
The goal of our experiments is to evaluate whether PFPN can outperform the corresponding implementations with Gaussian policies in data-driven control tasks of physics-based character.
We focus on the two aspects of comparisons: (1) the imitation ability of the trained policies, and (2) the policy robustness facing external force perturbation.
We also perform sensitivity analysis on the learning performance of PFPN with different number of particles and resampling strategies.
4.1 Setup
We run benchmarks based on the DeepMimic framework [Peng et al., 2018], which is the state-ofthe-art DRL imitation learning framework for physics-based character control.
We considered two [Figure]
R
[Figure]
w a r d s
Walk
[Figure]
[Figure]
articulated characters, a humanoid and a dog. The humanoid character has 8 spherical joints and 4 revolute joints, resulting in an observation space of R 197 and action space of R 36 .
The dog character has 18 spherical joints and 4 revolute joints, which lead to an observation space of R 301 and action space of R 76 .
All simulations are run using PyBullet [Coumans and Bai, 2016–2021].
PFPN focuses only on the distribution for action policy and can be applied with any general policy-gradient DRL algorithm.
In the following, we evaluate PFPN with policies trained using DPPO [Heess et al., 2017] with surrogate policy loss [Schulman et al., 2017]
We refer to Appendix E for results obtained with the
A3C
[Mnih et al., 2016] and IMPALA algorithms [Espeholt et al., 2018], and to Appendix D for all hyperparameters and the network architecture used.
4.2 Baseline Comparison
In Figure 2 and 3, we compare PFPN to baselines using Gaussian distribution with state-dependent mean and standard deviation values on the humanoid and dog character control tasks respectively.
The imitation performance is measured in the term of normalized cumulative rewards of evaluation rollouts during training.
We also compare PFPN to a ﬁxed discretization scheme (DISCRETE) obtained by uniformly discretizing each action dimension into a ﬁxed number of bins and sampling actions from a categorical distribution, and to policies using the distribution of a fully state-dependent
Gaussian Mixture Model (GMM).
All baselines use the same network architecture with the same number of hidden neurons.
PFPN, DISCRETE and GMM also have the same number of atomic actions (35 particles/bins) at each action dimension.
We train ﬁve trials of each baseline with different random seeds that are the same across PFPN and the corresponding implementations of other methods.
Evaluation was performed ten times every 1,000 training steps using deterministic actions.
As it can be seen from the ﬁgures, PFPN outperforms other baselines in all of the tested tasks.
Our particle-based scheme achieves better ﬁnal performance and exhibits faster convergence while being more stable across multiple trials.
GMM uses Gaussian components with state-dependent mean and standard deviation values, and does not work better than Gaussian baselines.
DISCRETE discretizes the action space using the atomic actions, which are the same with the initial distribution of the particles employed by PFPN.
Though both DISCRETE and PFPN use the state-independent atomic action settings, PFPN optimizes the distribution of atomic actions represented by particles during training, and the resulting adaptive discretization scheme leads to higher asymptotic performance.
Intuitively, by introducing more atomic actions, DISCRETE and PFPN could achieve higher control capacity.
However, the more atomic actions employed the harder the optimization problem will be due to the increase of policy gradient variance [Tang and Agrawal, 2019]
(see Appendix F for theoretical analysis).
In theory, GMM is a more general case of PFPN without resampling.
However,
[Figure]
[Figure]
X Axis -1.0 -0.6 -0.2 0.
Y Axis -1.0 -0.6 -0.2 0.
Z Axis -1.0 -0.6 -0.2 0.
Angle 1.5e7 T r a i n i n g S t e p s
[Figure]
Reference
PFPN
Gaussian
DISCRETE
GMM
[Figure]
we found that GMM does not usually work quite well and can perform even worse than Gaussians.
This is consistent with the results reported by Tang and Agrawal [2018] for torque-based locomotion control tasks.
A possible reason is that GMM needs a much larger policy network.
This may pose a challenge to the optimization.
Another issue of GMMs observed during our experiments is that the
Gaussian components are easy to collapse together and lose the advantage of multimodality.
To gain a better understanding of the advantages of using PFPN for learning motor tasks, Figureshows how particles evolve during training for one of the humanoid’s joints in the “Walk” task.
We can see that each of the ﬁnal active action spaces follows a multimodal distribution covering only some small parts of the entire space.
PFPN optimizes the placement of atomic actions, providing an effective discretization scheme that reaches better performance compared to other baselines.
4.3 Motion Quality
In Figure 5, we compare the foot trajectories of each baseline during ten gait cycles in “Walk” tasks.
From the ﬁgure, GMM generates jittery motions with unstable foot trajectories.
DISCRETE can provide relatively stable trajectories, in which, however, the character moves always towards an inclined direction.
Gaussian gives gaits with a cyclic pattern but not quite stable.
PFPN generate stable gaits with a clear cyclic pattern closely following the reference motion.
In Figure 6, we qualitatively compare the motions generated by PFPN and Gaussian baselines.
As can be seen, the motion generated by PFPN follows the reference motion (shadow character) closely.
Gaussian, though providing a human-like walking motion, exhibits visual artifacts in the other, more complex tasks, e.g. foot sliding and character drifting during dancing.
Similar conclusions can be drawn for the dog character.
When a ﬁxed number of samples are exploited for training as reported in Figure 3,
PFPN baselines can track the reference motion closely while Gaussian baselines perform worse, with the dog, for example, having evident jerky movements during pacing.
[Figure] [Figure]
average the
[Figure]
In addition, we note that characters trained with Gaussian policies often lack the grace seen in living beings as compared to PFPN-trained characters that exhibit behavior more close the reference data.
For example, while the dog is able to achieve a relatively high reward according to DeepMimic’s tracking reward function (Figure 3), its tail moves in an unnatural way during canter.
Similarly, while this is more subtle, the Gaussian-based humanoid stamps its feet on the ground while walking, resulting in a heavy-footed gait as compared to the graceful gait of the PFPN-based humanoid.
We refer to the supplemental video for more details on the visual quality of the trained controllers.
4.4 Robustness
We evaluate policy robustness through projectile testing and also external force perturbations.
During projectile testing, the character is controlled by the “walk” policy, and a cube projectile with side length of 0.2m and initial velocity of 0.2m/s is cast towards the torso of the character.
Figure reports the number of frames that a policy can control the character before falling down on the ground while varying the mass of the cube.
In Table 1, we also report the minimal force needed to push the character down in “walk” and “punch” tasks for the humanoid, and in “pace” and “canter” tasks for the dog character.
The DISCRETE and GMM results are skipped for the dog character, as the respective trained controllers are unable to make the character walk even in the absence of external disturbances.
All policies are obtained using DPPO algorithms without any projectile training or external force perturbation.
It is evident that the character controlled by PFPN are more robust to external disturbances and can sustain much higher forces than other baselines.
We also refer to the supplemental video for the visual performance of different baselines under projectile testing.
4.5 PFPN Results with SAC
[Figure]
In this section, we highlight
PFPN’s performance in state-ofthe-art off-policy DRL algorithm of SAC [Haarnoja et al., 2018c].
As shown in Figure 8, PFPN outperforms Gaussian baselines with faster convergence speed in all the tested tasks using SAC.
While SAC has been successfully explored for continuous control problems in the machine learning community, DPPO is still the most commonly used DRL algorithm for training physics-based character controllers in the animation ﬁeld.
During experiments, we found that PFPN with SAC is more stable in terms of learning performance and more sampling efﬁcient compared to DPPO implementations as reported in Figure 2.
Overall, PFPN with SAC needs only 2.5-million samples to train high-quality humanoid controllers in DeepMimic tasks as compared to DPPO that typically requires around 20 millions or even more samples to achieve the similar performance.
We refer to the supplementary video for comparisons between SAC and DPPO.
Discussion and Future Work
We present PFPN as a general framework for systematic exploration of high-dimensional action spaces during training of physics-based character controllers.
Our approach uses a mixture of state-independent Gaussians represented by a set of weighted particles to track the action policy, as opposite to the multivariate Gaussian that is typically used as the policy distribution in the tasks of physics-based character control.
We show that our method performs better than Gaussian baselines in various imitation learning tasks leading to faster learning, higher motion quality and more robustness to external perturbation.
In the experimental section, we showed applications of PFPN to the PPO algorithm.
However, our approach is applicable to other common on-policy actor-critic policy gradient DRL algorithms as we show in Appendix E and off-policy methods such as SAC, as we discussed in Section 4.5.
Overall,
PFPN does not change the underlying architecture or learning mechanism of the DRL algorithms.
It is, therefore, complementary to other techniques which improve the policy expressivity given a base action distribution.
For example, PFPN can serve as the action policy for each expert in the mixture of experts approach of [Won et al., 2020] or for primitive action learning [Peng et al., 2019]; or as the base distribution of normalizing ﬂows for motion generation [Henter et al., 2020], which we would like to investigate in future work.
Currently, we track each action dimension independently.
Accounting for the synergy that exists between different joints has the potential to further improve performance and motion robustness, which opens another exciting avenue for future work.
In any case, we believe that our particle-based action policies provide a great alternative to Gaussianbased actions policies, which have been the staple for DRL-based character control over the past few years.
Our work shows signiﬁcant improvements upon the state-of-the-art in terms of motion quality, robustness to external perturbations, and training efﬁciency for physics-based character control tasks.
B Policy Network Logits Correction during Resampling
Theorem.
Let D τ be a set of dead particles sharing the same target particle τ .
Let also the logits for the weight of each particle k be generated by a fully-connected layer with parameters ω k for the weight and b k for the bias. The policy π P θ ( a t | s t ) is guaranteed to remain unchanged after resampling via duplicating φ i ← φ τ , ∀ i ∈ D τ , if the weight and bias used to generate the unnormalized logits of the target particle are shared with those of the dead one as follows: [Equation]
Proof.
The weight for the i -th particle is achieved by softmax operation, which is applied to the unnormalized logits L i , which is the direct output of the policy network:
[Equation]
Resampling via duplicating makes dead particles become identical to their target particle.
Namely, particles in D τ ∪ { τ } will share the same weights as well as the same value of logits, say L (cid:48)
τ , after resampling.
To ensure the policy identical before and after sampling, the following equation must be satisﬁed
[Equation]
where L k is the unnormalized logits for the k -th particle such that the weights for all particles who are not in D τ ∪ { τ } unchanged, while particles in D τ ∪ { τ } share the same weights.
A target particle will not be tagged as dead at all, i.e. τ (cid:54)∈ D k for any dead particle set D k , since a target particle is drawn according to the particles’ weights and since dead particles are deﬁned as the ones having too small or zero weight to be chosen.
Hence, Equation 13 can be rewritten as (cid:88)
[Equation]
[Equation]
Given that e L i ( s t ) ≈ 0 for any dead particle i ∈ D τ and that the number of particles is limited, it implies that
[Equation]
Taking the logarithm of both sides of the equation leads to that for all particles in D τ ∪ { τ } , their new logits after resampling should satisfy
[Equation]
Assuming the input of the full-connected layer who generates L i is x
( s t ) ,
i
[Equation]
i ( s t
[Equation]
i
[Equation]
b
[Equation]
, we have
[Equation]
Then, Theorem can be reached.
If we perform unweighted resampling, it is possible to pick a dead particle as the target particle for some particles.
In that case
[Equation]
[Equation]
where L (cid:48) τ ( s t ) is the new logits shared by particles in D τ and δ ( τ, D k ) is the Kronecker delta function
[Equation]
that satisﬁes (cid:80) k
δ ( τ, D k ) ≤ 1 .
Then, for the particle τ , its new logits can be deﬁned as
[Equation]
Consequently, the target particle τ may or may not share the same logits with those in D τ , depending on if it is tagged as dead or not.
C PFPN with Off-Policy Policy Gradient Algorithms
To enable PFPN applicable in state-action value based off-policy algorithms, we propose a reparamterization trick in this section such that a sampled action a P θ ( s t ) can be differentiable to the policy network parameter θ .
C.1 Reparameterization Trick
Let x ( s t | θ ) ∼ C ONCRETE ( { w i ( s t | θ ); i = 1 , 2 , · · · } , λ ) is a sampling result of a relaxed version of the one-hot categorical distribution supported by the probability of { w i ( s t | θ ); i = 1 , 2 , · · · } , where x ( s t | θ )
=
{ x i ( s t | θ ); i = 1 , 2 , · · · } is reparametrizable and λ is picked to be 1 in our implementation.
We apply the Gumbel-softmax trick [Jang et al., 2017] to get a sampled action value as
[Equation]
where a i is the sample drawn from the distribution represented by the particle i with parameter φ i ,
STOP ( · ) is a “gradient stop” operation, and δ ( · , · ) denotes the Kronecker delta function.
Then, the reparameterized sampling result can be written as follows:
[Equation]
where m i :
= x i ( s t | θ )+ STOP ( δ ( i, arg max x ( s t | θ ))
− x i ( s t | θ ))
≡ δ ( i, arg max x ( s t | θ )) composing a one-hot vector that approximates the samples drawn from the corresponding categorical distribution.
Since x i ( s t | θ ) drawn from the concrete distribution is differentiable to the parameter θ , the gradient of the reparameterized action sample can be obtained by
[Equation]
Through these equations, both the policy network parameter θ and the particle parameters φ i can be updated by backpropagation through the sampled action a
[Equation]
( s
[Equation]
C.2 Policy Representation with Action Bounds
In off-policy algorithms, like DDPG and SAC, an invertible squashing function, typically the hyperbolic tangent function, will be applied to enforce action bounds on samples drawn from Gaussian distributions, e.g. in SAC, the action for the k -th dimension at the time step t is obtained by
[Equation]
(24) where u t,k ∼ N ( µ θ ( s t ) , σ 2 θ ( s t )) , µ θ ( s t ) and σ 2 θ ( s t ) are parameters generated by the policy network with parameter θ , and u t,k can be written u t,k
= µ θ ( s t ) + ξ t,k σ 2 θ ( s t ) given a noise variable ξ t,k ∼ N (0 , 1) such that a t,k is reparameterizable.
In our SAC implementation, we use Gaussian noises to generate action samples, i.e. u i,k ∼ N ( µ i,k , ξ 2 i,k ) where µ i,k and ξ i,k are the parameters for the i -th particle at the k -th action dimension.
Let a t = { tanh u t,k } where u t,k , drawn from the distribution represented by a particle with parameter φ t,k , is a random variable sampled to support the action on the k -th dimension.
Then, the probability density function of PFPN represented by Equation 3 can be rewritten as
[Equation]
and the log-probability function becomes log
[Equation]
=
[Equation]
D Hyperparameters
[Table]
, IMPALA) 0.
[Figure]
reply buffer size (SAC)
[Figure]
⊕ denotes the concatenation operator.
We set the resampling interval as 1200 environment steps, which are 20 environment episodes of simulation.
Since it is infeasible to analytically evaluate the differential entropy of a mixture distribution without approximation, we use the entropy of the categorical distribution for A3C and IMPALA benchmarks, which employ differential entropy during policy optimization.
In our implementation, policy and value networks have a similar structure of two hidden fully-connected (FC) layers with neurons of 1024 and 512 respectively, as shown in Figure 9.
The input state is normalized by moving average that is dynamically updated during training. By default, we place 35 particles on each action dimension and use the set of particles as a mixture of Gaussians with state-dependent weights but state-independent components. E Additional Results E.1 Time Complexity [Figure]
All policies were trained on a machine with Intel 6148G CPU and Nvidia V100 GPU.
Training stops when a ﬁxed number of samples is collected as reported in Figure 2.
PFPN has a good time consumption performance compared to other baselines.
Though the action sampling and particles resampling processes would take extra time, PFPN performs better because its fast convergence avoids wasting time on environment reset when early termination occurs.
E.2 Baselines
[Figure]
F Variance of Policy Gradient in PFPN Conﬁguration
Since each action dimension is independent to others, without loss of generality, we here consider the action a t with only one dimension along which n particles are distributed and the particle i to represent a Gaussian distribution N ( µ i , σ 2 i ) .
In order to make it easy for analysis, we set up the following assumptions: the reward estimation is constant, i.e. A t ≡ A ; logits to support the weights of particles are initialized equally, i.e. w i ( s t | θ ) ≡ 1 n for all particles i and ∇ θ w 1 ( s t | θ )
= · · · = ∇ θ w n ( s t | θ ) ; particles are initialized to equally cover the whole action space, i.e. µ i = i − nn , σ 2 i ≈ 1 n 2 where i
= 1 , · · · , n .
From Equation 7, the variance of the policy gradient under such assumptions is
[Equation]
Given V [ ∇ θ J ( θ ) | a t ] = 0 when n = 1 , from Equation 27, for any n > 0 , the variance of policy gradient V [ ∇ J ( θ ) | a t ] will increase with n .
Though the assumptions usually are hard to meet perfectly in practice, this still gives us an insight that employing a large number of particles may result in more challenge to optimization.
This conclusion is consistent with that in the case of uniform discretization [Tang and Agrawal, 2019] where the variance of policy gradient is shown to satisfy
[Equation]
[Equation]
That is to say, in either PFPN or uniform discretization scheme, we cannot simply improve the control performance of the police by employing more atomic actions, i.e. by increasing the number of particles or using more bins in the uniform discretization scheme, since the gradient variance increases as the discretization resolution increases.
However, PFPN has a slower increase rate, which implies that it might support more atomic actions before performance drops due to the difﬁculty in optimization.
Additionally, compared to the ﬁxed, uniform discretization scheme, atomic actions represented by particles in PFPN are movable and their distribution can be optimized.
This means that PFPN has the potential to provide better discretization scheme using fewer atomic actions to meet the ﬁne control demand and thus be more friendly to optimization using policy gradient.
G Ablation Study
Resampling Strategy.
We consider two resampling strategies as comparison: (1) the weighted resampling strategy described in Section 3.3 that draws targets for dead particles according to the weights of the remaining, alive ones, and (2) the unweighted resampling strategy that picks a τ i randomly from all alive particles.
In Figure 12a, we also perform comparison to PFPN without any resampling.
The weighted resampling strategy
The unweighted resampling strategy draws targets uniformly from alive particles.
It can be seen that both weighted and unweighted resampling could help improve the training performance signiﬁcantly.
However, unweighted resampling could lead to high variance by introducing too much uncertainty, since it could reactivate dead particles and place them in suboptimal locations with higher probability, compared to the weighted resampling.
After resampling, even though the particles would be optimized or resampled once more if they are placed in bad locations, this could make the training process converge slower, as shown in the ﬁgure.
Number of Particles.
Since the particle conﬁguration in PFPN is state-independent, it needs a sufﬁcient number of particles to meet the ﬁne control demand.
Intuitively, employing more particles will increase the resolution of the action space, and thus increase the control capacity and make
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
n
R
[Table]
w a r d
[Table]
Walk
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
ﬁne control more possible.
However, in Appendix F, we prove that due to the variance of policy gradient increasing as the number of particles increases, the more particles employed, the harder the optimization would be.
Therefore, it may negatively inﬂuence the performance to employ too many particles.
This conclusion is consistent with the results shown in Figure 12b.
As it can be seen, PFPN with 5 and 10 particles per action dimension performs badly; though using 50 or 100 particles per action dimension slows down the convergence speed a little bit, our approach is not sensitive in terms of the ﬁnal learning performance when a relative large number of particles are employed.