L EARNING WITH S OCIAL I NFLUENCE THROUGH I NTERIOR P OLICY D IFFERENTIATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Animals develop novel skills not only through the interaction with the environment but also from the inﬂuence of the others.
In this work we model the social inﬂuence into the scheme of reinforcement learning, enabling the agents to learn both from the environment and from their peers 1 .
Speciﬁcally, we ﬁrst deﬁne a metric to measure the distance between policies then quantitatively derive the definition of uniqueness.
Unlike previous precarious joint optimization approaches, the social uniqueness motivation in our work is imposed as a constraint to encourage the agent to learn a policy different from the existing agents while still solve the primal task.
The resulting algorithm, namely Interior Policy Differentiation (IPD), is able to learn a collection of policies that can solve a given task with distinct behaviors and brings about performance improvement as a byproduct in some cases.
I NTRODUCTION
The paradigm of Reinforcement Learning (RL), inspired by cognition and animal studies (Thorndike, 2017; Schultz et al., 1997), can be described as learning by interacting with the environment to maximize a cumulative reward (Sutton et al., 1998).
From the perspective of ecology, biodiversity as well as the development of various skills are crucial to the continuation and evolution of species (Darwin, 1859; Pianka, 1970).
Thus the behavioral diversity becomes a rising topic in RL.
Previous works have tried to encourage the emergence of behavioral diversity in RL with two approaches: The ﬁrst approach is to design interactive environments which contain sufﬁcient richness and diversity.
For example, Heess et al.
(2017) show that rich environments enable agents to learn different locomotion skills even using the standard RL algorithms.
Yet designing a complex environment requires manual efforts, and the diversity is limited by the obstacle classes.
The second approach to increase behavioral diversity is to motivate agents to explore beyond just maximizing the reward for the given task.
Zhang et al.
(2019) proposed to maximize a heuristically deﬁned novelty metric between policies through task-novelty joint optimization, but the ﬁnal performance of agents is not guaranteed.
In this work, we address the topic of policy differentiation in RL, i.e., to improve the diversity of
RL agents while keeping their ability to solve the primal task.
We draw the inspiration from the
Social Inﬂuence in animal society (Rogoff, 1990; Ryan & Deci, 2000; van Schaik & Burkart, 2011;
Henrich, 2017; Harari, 2014) and formulate the concept of social inﬂuence in the reinforcement learning paradigm.
Our learning scheme is illustrated in Fig 1.
The target agent not only learns to interact with the environment to maximize the reward but also differentiate the actions it takes in order to be different from other existing agents.
Since the social inﬂuence often acts on people passively as a sort of peer pressure, we implement the social inﬂuence in terms of social uniqueness motivation (Chan et al., 2012) and consider it as a constrained optimization problem.
In the following of our work, we ﬁrst deﬁne a rigorous policy distance metric in the policy space to compare the similarity of the agents.
Then we develop an optimization constraint using the proposed metric, which brings immediate rather than episodic feedback in the learning process.
A novel method, namely Interior Policy Differentiation (IPD), is further
[Figure]
Agent
Social Uniqueness
Primal Task
Run
A g e n t A g e n t
A g e n t
Agent
I should learn to run as fast as I can
I should try to be different proposed as a better solution for the constrained policy optimization problem. We benchmark our method on several locomotion tasks and show it can learn various diverse and well-behaved policies for the given tasks based on the standard Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017).
R ELATED W ORK
Intrinsic motivation methods.
The Variational Information Maximizing Exploration (VIME) method is designed by Houthooft et al.
(2016) to tackle the sparse reward problems.
In VIME, an intrinsic reward term based on the maximization of information gains is added to contemporary
RL algorithms to encourage exploration.
The curiosity-driven methods, proposed by Pathak et al.
(2017) and Burda et al.
(2018a) deﬁne intrinsic rewards according to prediction errors of neural networks.
i.e., when taking previous unseen states as inputs, networks trained with previous states will tend to predict with low accuracy, so that such prediction errors can be viewed as rewards.
Burda et al.
(2018b) proposed Random Network Distillation (RND) to quantify intrinsic reward by prediction differences between a ﬁxed random initialized network and another randomly initialized network trained with previous state information.
Liu et al.
(2019) proposed Competitive Experience
Replay (CER), in which they use two actors and a centralized critic, and deﬁned an intrinsic reward by the state coincidence of two actors.
The values of intrinsic rewards are ﬁxed to be ± 1 for the two actors separately.
All of those approaches leverage the weighted sum of the external rewards, i.e., the primal rewards provided by environments, and intrinsic rewards that provided by different heuristics.
A challenging problem is the trade-off between external rewards and intrinsic rewards.
The Task-Novelty Bisector (TNB) learning method introduced by Zhang et al.
(2019) aims to solve such problem by jointly optimize the extrinsic rewards and intrinsic rewards.
Speciﬁcally, TNB updates the policy in the direction of the angular bisector of the two gradients, i.e., gradients of the extrinsic and intrinsic objective functions.
However, the foundation of such joint optimization is not solid.
Besides, creating an extra intrinsic reward function and evaluating the novelty of states or policies always requires additional neural networks such as auto-encoders.
Thus extra computation expenses are needed (Zhang et al., 2019) .
Diverse behaviors from rich environments and algorithms.
Heess et al.
(2017) introduce the
Distributed Proximal Policy Optimization (DPPO) method and enable agents with simulated bodies to learn complex locomotion skills in a diverse set of challenging environments.
Although the learning reward they utilize is straightforward, the skills their policy learned are quite impressive and effective in traveling terrains and obstacles.
Their work shows that rich environments can encourage the emergence of different locomotion behaviors, but extra manual efforts are required in designing such environments.
The research of Such et al.
(2018) shows that different RL algorithms may converge to different policies for the same task.
The authors ﬁnd that algorithms based on policy gradient tend to converge to the same local optimum in the game of Pitfall, while off-policy and value-based algorithms are prone to learn sophisticated strategies.
On the contrary, in this paper, we are more interested in how to learn different policies through a single learning algorithm and learn the capability of avoiding local optimum.
Population-based novelty-seeking methods.
Pugh et al.
(2016) establish a standard framework for understanding and comparing different approaches to searching for quality diversity (QD).
Conti et al.
(2018) investigate adding novelty search (NS) and QD to evolution strategies (ES) to avoid local optima as well as achieve higher performance.
Lehman & Stanley (2011; 2008) conclude that deriving an open-ended search algorithm that operates without pressure towards the ultimate objective is possible, suggesting ignoring the objective may often beneﬁt the search itself.
The work of Wang et al.
(2019) yields a new kind of open-ended algorithm which indicates the solution to one environment might be a stepping stone to a new level of performance in another.
Such et al.
(2017) evolve a DNN with a population-based genetic algorithm (GA) for challenging RL tasks.
By improving the vanilla TRPO algorithm (Schulman et al., 2015), Kurutach et al.
(2018) maintains model uncertainty given the data collected from the environment via an ensemble of deep neural networks.
Q UANTIFYING THE D ISTANCE BETWEEN P OLICIES
To encourage the emergence of behavioral diversity in RL, we ﬁrst deﬁne a metric to measure the difference between policies, which is the foundation for the later algorithm we propose.
We denote the learned policies as { π θ i ; θ
i ∈ Θ , i = 1 , 2 , ... } , wherein θ i represents parameters of the i -th policy, Θ denotes the whole parameter space.
In the following, we omit π and denote a policy π θ i as θ i for simplicity unless stated otherwise.
3. D EFINITION
Mathematically, a metric should satisfy three important properties, namely the identity, the symmetry as well as the triangle inequality.
Deﬁnition
A metric space is an ordered pair ( M, d ) where M is a set and d is a metric on M , i.e., a function d : M × M → R such that for any x, y, z ∈ M , the following holds:
d ( x, y ) ≥ 0 , d ( x, y )
=
⇔ x = y , 2.
d ( x, y ) =
d ( y, x ) , 3.
d ( x, z ) ≤ d ( x, y ) + d ( y, z ) .
We use the Total Variance Divergence D TV (Schulman et al., 2015) to measure the distance between policies.
Concretely, for discrete probability distributions p and q , this distance is deﬁned as
D TV ( p, q ) =
(cid:80) i | p i − q i | .
Theorem 1 (Metric Space (Θ , D ρTV ) )
The expectation of D TV ( · , · ) of two policies over any state distribution ρ ( s ) :
[Equation]
is a metric on Θ , thus (Θ , D ρTV ) is a metric space.
The proof of Theorem 1 is in Appendix A.
It is worth mentioning that, although TVD is used in our work, we can easily extend the result to use other distance between distributions as substitutes of TVD (e.g. Jensen Shannon divergence D JS or Wasserstein metric D W ) (Endres & Schindelin, 2003; Fuglede & Topsoe, 2004; Villani, 2008), and similar results can be get
Corollary 1 Let D ρJS :=
E s ∼ ρ ( s ) [ D JS ( θ i ( s ) , θ j ( s ))] and D ρW :=
E s ∼ ρ ( s ) [ D W ( θ i ( s ) , θ j ( s ))] , (Θ , D ρJS ) and (Θ , D ρW ) are also metric spaces.
On top of the metric space (Θ , D ρTV ) , we could then compute the uniqueness of a policy.
Deﬁnition 2 (Uniqueness of Policy)
Given a reference policy set Θ ref such that Θ ref = { θ ref i , i = 1 , 2 , ... } , Θ ref ⊂ Θ , the uniqueness U( θ | Θ ref ) of policy θ is the minimal difference between θ and all policy in the reference policy set, i.e.,
[Equation]
Consequently, to motivate RL with the social uniqueness, we hope our method can maximize the uniqueness of a new policy, i.e., max θ U( θ | Θ ref ) , where the Θ ref includes all the existing policies.
3. E STIMATION OF D ρTV ( θ i , θ j )
In practice, the calculation of D ρTV ( θ i , θ j ) is based on Monte Carlo estimation.
i.e., we need to sample s from ρ ( s ) .
Although in ﬁnite state space we can get precise estimation after establishing ergodicity, problem arises when we are facing continuous state cases.
i.e. it is difﬁcult to efﬁciently get enough samples.
Formally, we denote the domain of ρ ( s ) as S and denote the domain of ρ θ ( s ) as S θ ⊂ S , where ρ θ ( s ) :
= ρ ( s | s ∼ θ ) and in ﬁnite time horizon problems ρ ( s | s ∼ θ ) =
P ( s 0 = s | θ )
+ P ( s 1 = s | θ )+ ...
+ P ( s T = s | θ ) .
As we only care about the reachable regions, the domain S can be divided by S = lim N →∞ (cid:83)
Ni =1 S θ i .
In order to improve the sample efﬁciency, we propose to approximate D ρTV ( θ i , θ j ) with
D ρ θ TV ( θ i , θ j ) , where θ is a certain ﬁxed behavior policy that irrelevant to θ i , θ j .
Such approximation requires a necessary condition:
Condition 1 The domain of possible states are similar between different policies:
[Equation]
When such condition holds, we can use ρ ( s | s ∼ θ ) as our choice of ρ ( s ) , and the properties in
Deﬁnition 1 still holds.
In practice, the Condition 1 always holds as we can ensure this by adding sufﬁciently large noise on θ , while the permitted state space is always limited.
And for more general cases, to satisfy the properties in Deﬁnition 1, we must sample s from S θ ∪ S θ j , accordingly,
[Equation]
where N represents random action when a policy have never been trained or visited such state domain
Plugging Eq.(4) into Eq.(2), the objective function of policy differentiation is
[Equation]
While the ﬁrst two terms are related to the policy θ , the last term is only related to the domain S θ .
If we enable sufﬁcient exploration in training as well as in the initialization of θ , the last term will disappear (i.e. S θ j ⊂ S θ ).
Hence we can also use D ρ θi TV ( θ i , θ j ) as an approximation of D ρTV ( θ i , θ j ) in training of θ i as long as sufﬁcient exploration is guaranteed.
Proposition 1 (Unbiased Single Trajectory Estimation)
The estimation of ρ θ ( s ) using a single trajectory τ is unbiased.
The proof of Proposition 1 is in Appendix B.
Given the deﬁnition of uniqueness and a practically unbiased sampling method, the next step is to develop an efﬁcient learning algorithm.
I NTERIOR P OLICY D IFFERENTIATION
In the traditional RL paradigm, maximizing the expectation of cumulative rewards g = (cid:80) t =0 γ t r t is commonly used as the objective.
i.e. max θ ∈ Θ E τ ∼ θ [ g ] , where τ ∼ θ denotes a trajectory τ sampled from the policy θ using Monte Carlo methods.
To improve the behavioral diversity of different agents, the learning objective must take both reward from the primal task and the policy uniqueness into consideration.
Previous approaches (Houthooft et al., 2016; Pathak et al., 2017; Burda et al., 2018a;b; Liu et al., 2019) often directly write the weighted sum of the reward from the primal task and the intrinsic reward g int =
(cid:80) t =
γ t r int ,t , where r int ,t denotes the intrinsic reward (e.g., r int =
min θ j ∈ Θ ref D ρTV ( θ, θ j ) as the uniqueness reward in our case) as follows,
[Equation]
where 0 < α < 1 is a weight parameter.
Such an objective is sensitive to the selection of α as well as the formulation of r int .
For example, in our case formulating the intrinsic reward r int as min
[Equation]
D [Equation] θ, θ
[Equation]
, exp [min
[Equation]
D [Equation] ( θ, θ j )] and − exp [ − min θ j D ρTV ( θ, θ j )] will result in signiﬁcantly different results.
Besides, a trade-off arises in the selection of α : while a large α may undermine the contribution of intrinsic reward, a small α could ignore the importance of the reward, leading to the failure of agent in solving the primal task.
To tackle these issues, we draw inspiration from the observation that social uniqueness motivates people in passive ways.
In other words, it plays more like a constraint rather than an additional target.
Therefore, we change the multi-objective optimization problem in Eq.(6) into a constrained optimization problem as:
[Equation]
where r 0 is a threshold indicating minimal permitted uniqueness, and r int ,t denotes a moving average of r int ,t . Further discussion on the selection of r 0 will be deliberated in Appendix D.
From the perspective of optimization, Eq.(6) can be viewed as a penalty method which replaces the constrained optimization problem in Eq.(7) with the penalty term r int and the penalty coefﬁcient 1 − αα > 0 , where the difﬁculty lies in the selection of α .
The work of Zhang et al.
tackles this challenge by the Task Novel Bisector (TNB) in the form of Feasible Direction Methods (FDMs) (Zoutendijk, 1960).
As a heuristic approximation, that approach requires reward shaping and intensive emphasis on r int ,t .
Instead, in this work we propose to solve the constrained optimization problem Eq.(7) by resembling the Interior Point Methods (IPMs) (Potra & Wright, 2000;
Dantzig & Thapa, 2006).
In vanilla IPMs, the constrained optimization problem in Eq.(7) is solved by reforming it to an unconstrained form with an additional barrier term in the objective as
[Equation]
The limit of Eq.(8) when α → 0 then leads to the solution of Eq.(7). Readers please refer to
Appendix G for more discussion on the correspondence between those novel policy seeking methods and constrained optimization methods.
However, directly applying the IPMs is computationally challenging and numerically unstable, especially when α is small.
Luckily, in our proposed RL paradigm where the behavior of an agent is inﬂuenced by its peers, a more natural way can be used.
Precisely, since the learning process is based on sampled transitions, we can simply bound the collected transitions in the feasible region by permitting previous trained M policies θ i ∈ Θ ref , i = 1 , 2 , ..., M sending termination signals during the training process of new agents.
In other words, we implicitly bound the feasible region by terminating any new agent that steps outside it.
Consequently, during the training process, all valid samples we collected are inside the feasible region, which means these samples are less likely to appear in previously trained policies.
At the end of the training, we then naturally obtain a new policy that has sufﬁcient uniqueness.
In this way, we no longer need to consider the trade-off problem between intrinsic and extrinsic rewards deliberately.
The learning process of our method is thus more robust and no longer suffer from objective inconsistency.
As our formulation of the constrained optimization problem Eq.(7) is inspired by IPMs, we name our approach as Interior Policy
Differentiation (IPD) method.
E XPERIMENTS
The MuJoCo environment We demonstrate our proposed method on the OpenAI Gym where the physics engine is based on MuJoCo (Brockman et al., 2016;
Todorov et al., 2012).
Concretely, we test on three locomotion environments, the Hopper-v3 (11 observations and 3 actions), Walker2dv3 (11 observations and 2 actions), and HalfCheetah-v3 (17 observations and 6 actions).
In our experiments, all the environment parameters are set as default values.
Uniqueness beyond intrinsic stochasticity Experiments in Henderson et al.
(2018) show that policies that perform differently can be produced by simply selecting different random seeds before training.
Before applying our method to improve behavior diversity, we ﬁrstly benchmark how much uniqueness can be generated from the stochasticity in the training process of vanilla RL algorithms as well as the random weight initialization.
In this work, we mainly demonstrate our proposed method based on PPO(Schulman et al., 2017).
The extension to other popular algorithms is straightforward.
We also compare our proposed method with the TNB and weighted sum reward (WSR) approaches as different ways to combine the goal of the task and the uniqueness motivation (Zhang et al., 2019).
More implementation details are depicted in Appendix D.
5. U NIQUENESS AND P ERFORMANCE C OMPARISON
According to Theorem 2, the uniqueness r int in equation (7) under our uniqueness metric can be unbiased approximated by r int = min θ j ∈ Θ ref D ρ θ TV ( θ ( s t ) , θ j ( s t )) .
i.e., we utilize the metric directly in learning new policies instead of applying any kind of reshaping.
We implement WSR, TNB, and our method in the same experimental settings and for each method, 10 different policies are trained and try to be unique with regard to all previously trained policies
[Table]
sequentially.
Concretely, the 1 st policy is trained by ordinary PPO without any social inﬂuence.
The 2 nd policy should be different from 1 st policy [Table] and the 3 rd should be different from the previous two policies, and so on.
Fig.2 shows the qualitative results of our method.
We visualize the motion of agents by drawing multiple frames representing the pose of agents at different time steps in the same row.
The horizontal interval between consecutive frames is proportional to the velocity of agents.
The settings of the frequency of highlighted frames and the correlation between interval and velocity are ﬁxed for each environment.
The visualization starts from the beginning of each episode and therefore the readers can get sense of the process of acceleration as well as the pattern of motion of agents clearly.
Fig.
3 shows our experimental results in terms of uniqueness (the x-axis) and the performance (the y-axis).
Policies in the upper right are the more unique ones with higher performance.
In Hopper and HalfCheetah, our proposed method distinctively outperforms other methods.
In Walker2d, both
WSR and our method work well in improving the uniqueness of policies, but none of the three methods can ﬁnd way to surpass the performance of PPO apparently.
Detailed comparison on the task related rewards are carried out in Table 1.
A box ﬁgure depicting the performance of each trained policy and their reward gaining curve are disposed in Fig.5 and Fig.6 in Appendix C. And
Fig.7 in Appendix C provides more detailed results from the view of uniqueness.
5. S UCCESS R ATE OF E ACH M ETHOD
In addition to averaged reward, we also use success rate as another metrics to compare the performance of different approaches.
In this work, we consider a policy is success when its performance is at least as good as the averaged performance of policies trained without social inﬂuences.
To be speciﬁc, we use the averaged ﬁnal performance of PPO as the baseline.
If a new policy, which aims at performing differently to solve the same task, surpasses the baseline during its training process, it will be regarded as a successful policy.
Through the success rate, we know the policy does not learn unique behavior at the expense of performance.
Table 1 shows the success rate of all the methods, including the PPO baseline.
The results show that our method can always surpass the average baseline during training.
Thus the performance of our method can always be insured.
[Figure]
5. B ETTER P OLICY D ISCOVERY
In our experiments, we observed noticeable performance improvements in the Hopper and the
HalfCheetah environments.
For the environment of Hopper, in many cases, the agents trained with
PPO tend to learn a policy that jumps as far as possible and then fall to the ground and terminate this episode (please refer to Fig.11 in Appendix E).
Our proposed method can prevent new policies from always falling into the same local minimum.
After the ﬁrst policy being trapped in a local minimum, the following policies will try other approaches to avoid the same behavior, explore other feasible action patterns, and thereafter the performance may get improved.
Such property shows that our method can be a helpful enhancement of the traditional RL scheme, which can be epitomized as policies could make mistakes, but they should explore more instead of hanging around the same local minimum.
The similar feature attributes to the reward growth in the environment of
HalfCheetah.
Moreover, we can illuminate the performance improvement of HalfCheetah from another perspective.
The environment of HalfCheetah is quite different from the other two for there is no explicit termination signal in its default settings (i.e., no explicit action like falling to the ground would trigger termination).
At the beginning of the learning process, an agent will act randomly, resulting in massive repeat, trivial samples as well as large control costs.
In our learning scheme, since the agent also interacts with the peers, it can receive termination signals from the peers to prevent wasting too much effort acting randomly.
During the learning process in our method, an agent will ﬁrst learn to terminate itself as soon as possible to avoid heavy control costs by imitating previous policies and then learns to behave differently to pursue higher reward.
From this point of view, such learning process can be regarded as a kind of implicit curriculum.
5. S CALE OF THE I NFLUENCE
As the number of policies learned with social inﬂuence grows, the difﬁculty of ﬁnding a unique policy may also increase.
Later policies must keep away from all previous solutions.
The results of our ablation study on how the performance changes under different scales of social inﬂuence (i.e., the number of peers) is shown in Fig.
4, where the thresholds are selected according to our previous ablation study in Sec.
D. The performance decrease is more obvious in Hopper than the other two environments for the action space of Hopper is only 3 dimensional.
Thus the number of possible diverse policies can be discovered is limited.
C ONCLUSION
In this work, we develop an efﬁcient approach to motivate RL to learn diverse strategies inspired by social inﬂuence.
After deﬁning the distance between policies, we introduce the deﬁnition of policy uniqueness.
Regarding the problem as constrained optimization problem, our proposed method,
Interior Policy Differentiation (IPD), draws the key insight of the Interior Point Methods.
And our experimental results demonstrate IPD can learn various well-behaved policies, and our approach can help agents to avoid local minimum and can be interpreted as a kind of implicit curriculum learning in certain cases.
A P ROOF OF T HEOREM
The ﬁrst two properties are obviously guaranteed by D ρTV .
As for the triangle inequality,
E s ∼ ρ ( s ) [ D TV ( θ i ( s ) , θ k
( s )]
=
E s ∼ ρ ( s ) [ |A| (cid:88) l
= | θ i ( s ) − θ k
( s ) | ] =
E s ∼ ρ ( s ) [ |A| (cid:88)
l
= | θ i ( s ) − θ j
( s ) + θ j ( s ) − θ k
( s ) | ] ≤ E s ∼ ρ ( s ) [ ( |A| (cid:88) l
= | θ i ( s ) − θ j ( s ) | + | θ j ( s ) − θ k
( s ) | )]
=
E s ∼ ρ ( s ) [ |A| (cid:88) l
= | θ i ( s ) − θ j ( s ) | ] + E s ∼ ρ ( s ) [ |A| (cid:88) l = | θ j ( s ) − θ k
( s ) | ] =
E s ∼ ρ ( s ) [ D TV ( θ i ( s ) , θ j ( s )]
+ E s ∼ ρ ( s )
[ D TV ( θ j ( s ) , θ k
( s )]
B P ROOF OF P ROPOSITION
[Table]
=
[Table] [Table]
[Table]
+ (cid:80)
[Table]
=
[Table] [Table]
=
[Table]
τ
[Table]
N + ... + (cid:80)
[Table]
=
[Table]
I ( s T =
[Table]
τ
[Table]
N
[Table]
N
[Table]
[Equation]
N
[Equation]
C D ETAILS OF U NIQUENESS AND P ERFORMANCE
[Figure]
[Figure] [Figure]
[Figure] [Figure]
10 Number of policies 0.
U n i q u e n e ss
Walker-v 2 3 4 5 6 7 8 9 10 Number of policies 1.
U n i q u e n e ss
HalfCheetah-v
D I MPLEMENTATION D ETAILS Calculation of D TV
We use deterministic part of policies in the calculation of D TV , i.e., we remove the Gaussian noise on the action space in PPO and use D TV ( a 1 , a 2 ) =
| a 1 − a 2 | .
Network Structure We use MLP with 2 hidden layers as our actor models in PPO.
The ﬁrst hidden layer is ﬁxed to have 32 units.
Our ablation study on the choice of unit number in the second layer is detailed in Table.2, Table3 and Fig.8.
Moreover, we choose to use 10 , 64 and 256 hidden units for the three tasks respectively in all of the main experiments, after taking the success rate (Table.2), performance (Table.3) and computation expense (i e. the preference to use less unit when the other two factors are similar) into consideration.
[Figure]
U n i q u e n e ss i n t e r m s o f
T o t a l V a r i a n c e D i v e r g e n c e
[Figure] [Figure]
256hidden
[Figure]
−
[Figure]
− 1 .
[Figure]
[Table]
Timesteps We ﬁx the training timesteps in our experiments.
timesteps are ﬁxed to be 1 M in Hopper-v3, 1 . 6 M for Walker2d-v3 and 3 M for HalfCheetah.
[Table]
Threshold Selection
In our proposed method, we can control the magnitude of policy uniqueness ﬂexibly by adjusting the constraint threshold r 0 .
Choosing different thresholds will lead to different policy behaviors.
Concretely., a larger threshold may drive the agent to perform more differently while smaller threshold imposes a lighter constraint on the behavior of the agent.
Intuitively, a larger threshold will lead to relatively poor performance for the learning algorithm is less likely to ﬁnd a feasible solution to Eq.(7).
Besides, we do not use constraints in the form of Eq.(7) as we need not force every single action of a new agent to be different from others.
Instead, we are more care about the long term differences.
Therefore, we use the cumulative uniqueness as constraints,
[Equation]
We test our method with different choices of threshold values.
The performance of agents under different thresholds are shown in Fig.
9 and more detailed analysis of their success rate is presented in Table. [Figure] [Equation] [Figure]
[Equation]
[Figure]
E M ORE Q UALITATIVE R ESULTS
F I MPLEMENTATION OF E Q .(7)
We do not use constraints in the form of Eq.(7) as we need not force every single action of a new agent to be different from others.
Instead, we are more care about the long term differences.
Therefore, we use the cumulative uniqueness as constraints.
Moreover, the constraints can be applied after the ﬁrst t S timesteps (e.g. t S = 20 ) for the consideration of similar starting sequences.
[Equation]
G R ELATION BETWEEN D IFFERENT A PPROACHES AND C ONSTRAINED O PTIMIZATION M ETHODS
We note here, the WSR, TNB and IPD methods correspond to three approaches in constrained optimization problem.
For simplicity, we consider Eq.(9) with a more concise notion g int ,t − g 0 ,t ≥ 0 , where g int ,t = (cid:80) tt =0 r int ,t , i.e.,
[Equation]
As the optimization of policy is based on batches of trajectory samples and is implemented with stochastic gradient descent, Eq.(10) can be further simpliﬁed as:
[Equation]
where g t ( θ ) denotes the average over a trajectory.
WSR: Penalty Method The Penalty Method considers the constraints of Eq.(11) by putting constraint g ( θ ) into a penalty term, and then solve the unconstrained problem
[Equation]
using an iterative manner, and the limit when α → 0 lead to the solution of the primal constrained problem.
As an approximation, WSR choose a ﬁxed weight term α , and use the gradient of ∇ θ f + 1 − αα ∇ θ g instead of ∇ θ f + 1 − αα ∇ θ min { g ( θ ) , 0 } , thus the ﬁnal solution will intensely rely on the selection of α .
TNB:
Feasible Direction Method The Taylor series of g ( θ ) at point ¯ θ is
[Equation]
The Feasible Direction Method (FDM) considers the constraints of Eq.(11) by ﬁrst ﬁnding a direction (cid:126)p satisﬁes
[Equation]
so that for small λ , we have
[Equation]
[Equation]
and
[Equation]
[Equation]
The TNB method, by using the bisector of gradients ∇ θ f and ∇ θ g , select (cid:126)p to be
[Equation]
Clearly, Eq.(17) satisﬁes Eq.(14), but it is more strict than Eq.(14) as the ∇ θ g term always exists during the optimization of TNB.
In TNB, the learning stride is ﬁxed to be |∇ θ f | + |∇ θ g |, leading to problem when ∇ θ f → 0 , which shows the ﬁnal optimization result will heavily rely on the selection of g .
i.e., the shape of g is crucial for the success of TNB.
IPD: Interior Point Methods (IPMs) In vanilla IPMs, the constrained optimization problem in
Eq.(11) is solved by reforming it to an unconstrained form with an additional barrier term α
[Equation]
θ ) in the objective as
[Equation]
18) or use the barrier term of − α log g ( θ ) instead:
[Equation]
) where α , the barrier factor, is a small positive number.
As α is small, the barrier term will introduce only minuscule inﬂuence on the objective.
On the other hand, when θ get closer to the barrier, the objective will increase fast.
It is clear that the solution of the objective with barrier term will get closer to the primal objective as α getting smaller.
Thus in practice, such methods will choose a sequence of { α i } such that 0 <
α i < α k +1 and α i → 0 as k → ∞
The limit of Eq.(18), Eq.(19) when α → 0 then leads to the solution of Eq.(11).
The work of Conn et al.
; Wright (2001) provide proofs of the convergence.
Directly applying this method is computationally challenging and numerically unstable, especially when α is small.
A more natural way can be used: since the learning process is based on sampled transitions, we can simply bound the collected transitions in the feasible region by permitting previous trained M policies θ i ∈ Θ ref , i = 1 , 2 , ..., M sending termination signals during the training process of new agents.
In other words, we implicitly bound the feasible region by terminating any new agent that steps outside it.
Consequently, during the training process, all valid samples we collected are inside the feasible region, which means these samples are less likely to appear in previously trained policies.
At the end of the training, we then naturally obtain a new policy that has sufﬁcient uniqueness.
In this way, we no longer need to consider the trade-off problem between intrinsic and extrinsic rewards deliberately.
The learning process of our method is thus more robust and no longer suffer from objective inconsistency.
Algorithm.1 shows the pseudo code of IPD based on PPO, where the blue lines show the addition to primal PPO algorithm.
• a behavior policy θ old • a set of previous policies { j } , j = 1 , 2 , ..., M • a uniqueness metric U ( θ, { θ j }| ρ ) =
U ( θ, { θ j }| τ ) =
min θ j D τTV ( θ, θ j ) • a uniqueness threshold r 0 , starting point t S Initialize
for do do
for t =
2 , ..., T
Run policy θ old in environment, get trajectory τ
if then done =
True end if if done then break end if end for
Compute advantage estimates ˆ A
A end for
Optimize surrogate L CLIP w.r.t.
θ , with K epochs and minibatch size M ≤ NT θ old ← θ end for