Original: C OMPOSING E NTROPIC P OLICIES USING D IVERGENCE
Revised: C OMPOSING E NTROPIC P OLICIES USING D IVERGENCE

Original: C
Revised: C

Original: ORRECTION
Revised: ORRECTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep reinforcement learning (RL) algorithms have made great strides in recent years.
Revised: Deep reinforcement learning (RL) algorithms have made great strides in recent years.

Original: An important remaining challenge is the ability to quickly transfer existing skills to novel tasks, and to combine existing skills with newly acquired ones.
Revised: An important remaining challenge is the ability to quickly transfer existing skills to novel tasks, and to combine existing skills with newly acquired ones.

Original: In domains where tasks are solved by composing skills this capacity holds the promise of dramatically reducing the data requirements of deep RL algorithms, and henceof greatly increasing their applicability.
Revised: In domains where tasks are solved by composing skills this capacity holds the promise of dramatically reducing the data requirements of deep RL algorithms, and hence increasing their applicability.

Original: Recent work has studied ways of composing behaviors represented in the form of action-value functions.
Revised: Recent work has studied ways of composing behaviors represented in the form of action-value functions.

Original: We analyze these methods to highlight their strengths and weaknesses, and point out situations where each of them is susceptible to poor performance.
Revised: We analyze these methods to highlight their strengths and weaknesses, and point out situations where each of them is susceptible to poor performance.

Original: To perform this analysis we extend generalized policy improvement to the max-entropy framework and introduce a method for the practical implementation of successor features in continuous action spaces.
Revised: To perform this analysis we extend generalized policy improvement to the max-entropy framework and introduce a method for the practical implementation of successor features in continuous action spaces.

Original: Then we propose a novel approach which achieves an approximately optimal result.
Revised: Then we propose a novel approach which, in principle, recovers the optimal policy during transfer.

Original: This method works by explicitly learning the (discounted, future) divergence between policies.
Revised: This method works by explicitly learning the (discounted, future) divergence between policies.

Original: We study this approach in the tabular case and propose a scalable variant that is applicable in multi-dimensional continuous action spaces.
Revised: We study this approach in the tabular case and propose a scalable variant that is applicable in multi-dimensional continuous action spaces.

Original: We compare our approach with existing ones on a range of non-trivial continuous control problems with compositional structure, and demonstrate near-optimal performance despite  requiring less information than competing approaches.
Revised: We compare our approach with existing ones on a range of non-trivial continuous control problems with compositional structure, and demonstrate qualitatively better performance despite not requiring simultaneous observation of all task rewards.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning algorithms coupled with powerful function approximators have recently achieved a series of successesplaying video games (Mnih et al., 2015) or Go (Silver et al., 2016), as well as in simulated and real robotic control (Lillicrap et al., 2015;
Revised: Reinforcement learning algorithms coupled with powerful function approximators have recently achieved a series of successes (Mnih et al., 2015; Silver et al., 2016; Lillicrap et al., 2015;

Original: Kalashnikov et al., 2018).
Revised: Kalashnikov et al., 2018).

Original: Unfortunately, while being extremely powerful deep reinforcement learning (DRL) algorithms often require a large number of interactions with the environment to achieve good results, partially because they are often applied “from scratch” rather than in settings where they can leverage existing experience.
Revised: Unfortunately, while being extremely powerful, deep reinforcement learning (DRL) algorithms often require a large number of interactions with the environment to achieve good results, partially because they are often applied “from scratch” rather than in settings where they can leverage existing experience.

Original: This reduces their applicability in domains where generating experience is expensive, or learning from scratch is challenging.
Revised: This reduces their applicability in domains where generating experience is expensive, or learning from scratch is challenging.

Original: The data efﬁciency of DRL algorithms is affected by various factors and signiﬁcant research effort has been directed at achieving improvements (e.g. Popov et al., 2017).
Revised: The data efﬁciency of DRL algorithms is affected by various factors and signiﬁcant research effort has been directed at achieving improvements (e.g. Popov et al., 2017).

Original: At the same time the development of basic locomotor behavior in humans can, in fact, require large amounts of experience and practice (Adolph et al., 2012), and it can take signiﬁcant effort and training to master complex, high-speed skills (Haith & Krakauer, 2013).
Revised: At the same time the development of basic locomotor behavior in humans can, in fact, require large amounts of experience and practice (Adolph et al., 2012), and it can take signiﬁcant effort and training to master complex, high-speed skills (Haith & Krakauer, 2013).

Original: Once such skills have been acquired, however, humans rapidly put them to work in new contexts and to solve new tasks, suggesting transfer learning as an important mechanism.
Revised: Once such skills have been acquired, however, humans rapidly put them to work in new contexts and to solve new tasks, suggesting transfer learning as an important mechanism.

Original: Transfer learning has been explored extensively in multiple ﬁelds of the machine learning community (see e.g. Weiss et al., 2016, for a recent review).
Revised: Transfer learning has been explored extensively in multiple ﬁelds of the machine learning community (see e.g. Weiss et al., 2016, for a recent review).

Original: For instance, in the computer vision literature it is common practice to transfer experience in the form of visual feature vectors(Huh et al., 2016;

Original: Kornblith et al., 2018).

Original: In RL and robotics the transfer of knowledge from one task to another has been studied from a variety of angles.
Revised: In RL and robotics the transfer of knowledge from one task to another has been studied from a variety of angles.

Original: For the purpose of this paper we are interested in methods that are suitable for transfer in the context of high-dimensional motor control problems.
Revised: For the purpose of this paper we are interested in methods that are suitable for transfer in the context of high-dimensional motor control problems.

Original: We further focus on model-free approaches, which are evident in human motor control (Haith & Krakauer, 2013), and have recently been used by a variety of scalable deep RL methods (e.g. Lillicrap et al., 2015; Mnih et al., 2015; 
Revised: We further focus on model-free approaches, which are evident in human motor control (Haith & Krakauer, 2013), and have recently been used by a variety of scalable deep RL methods (e.g. Lillicrap et al., 2015; Mnih et al., 2015; Schulman et al., 2017;

Original: Schulman et al., 2017; Kalashnikov et al., 2018).
Revised: Kalashnikov et al., 2018).

Original: Transfer may be especially valuable in domains where a small set of skills can be composed, in different combinations, to solve a variety of tasks.
Revised: Transfer may be especially valuable in domains where a small set of skills can be composed, in different combinations, to solve a variety of tasks.

Original: Different notions of compositionality have been considered in the RL and robotics literature.
Revised: Different notions of compositionality have been considered in the RL and robotics literature.

Original: For instance, ‘options’ are associated with discrete units of behavior that can be sequenced, thus emphasizing composition in time (Precup et al., 1998).
Revised: For instance, ‘options’ are associated with discrete units of behavior that can be sequenced, thus emphasizing composition in time (Precup et al., 1998).

Original: In this paper we are concerned with a rather distinct notion of compositionality, namely how to combine and blend potentially concurrent behaviors.
Revised: In this paper we are concerned with a rather distinct notion of compositionality, namely how to combine and blend potentially concurrent behaviors.

Original: This form of composition is particularly relevant in high-dimensional continuous action spaces, where it is possible to achieve more than one task simultaneously (e.g. walking somewhere while juggling).
Revised: This form of composition is particularly relevant in high-dimensional continuous action spaces, where it is possible to achieve more than one task simultaneously (e.g. walking somewhere while juggling).

Original: One approach to this challenge is via the composition of task rewards.
Revised: One approach to this challenge is via the composition of task rewards.

Original: Speciﬁcally, we are interested in the following question: If we have previously solved a set of tasks with similar transition dynamics but different reward functions, how can we leverage this knowledge to solve new tasks which can be expressed as a convex combination of those rewards functions?
Revised: Speciﬁcally, we are interested in the following question: If we have previously solved a set of tasks with similar transition dynamics but different reward functions, how can we leverage this knowledge to solve new tasks which can be expressed as a convex combination of those rewards functions?

Original: This question has recently been studied in two independent lines of work: by Barreto et al.
Revised: This question has recently been studied in two independent lines of work: by Barreto et al.

Original: (2017; 2018) in the context of successor feature (SF) representations used for Generalized Policy Improvement (GPI) with deterministic policies, and by Haarnoja et al.
Revised: (2017; 2018) in the context of successor feature (SF) representations used for Generalized Policy Improvement (GPI) with deterministic policies, and by Haarnoja et al.

Original: (2018a); van Niekerk et al.
Revised: (2018a); van Niekerk et al.

Original: (2018) in the context of maximum entropy policies.
Revised: (2018) in the context of maximum entropy policies.

Original: These approaches operate in distinct frameworks but both achieve skill composition by combining the Q -functions associated with previously learned skills.
Revised: These approaches operate in distinct frameworks but both achieve skill composition by combining the Q -functions associated with previously learned skills.

Original: We clarify the relationship between the two approaches and show that both can perform well in some situations but achieve poor results in others, often in complementary ways.
Revised: We clarify the relationship between the two approaches and show that both can perform well in some situations but achieve poor results in others, often in complementary ways.

Original: We introduce a novel method of behavior composition that that can consistently achieve good performance.
Revised: We introduce a novel method of behavior composition that that can consistently achieve good performance.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Revised: We introduce succcessor features (SF) in the context of maximum entropy and extend the GPI theorem to this case (max-ent GPI).

Original: We provide an analysis of when GPI, and “optimistic” compositions of entropy-regularized policies (Haarnoja et al., 2018a)  transfer.
Revised: We provide an analysis of when GPI, and compositional “optimism” (Haarnoja et al., 2018a) of entropy-regularized policies transfer.

Original: We construct both tabular and continuous action tasks where both fail to transfer well.
Revised: We construct both tabular and continuous action tasks where both fail to transfer well.

Original: We introduce succcessor features (SF) in the context of maximum entropy and extend GPI to this case (max-ent GPI).

Original: We propose a correction term – which we call Divergence Correction (DC)– based on the R ´ enyi divergence between policies which allows us, in principle, to recover the optimal policy for transfer for any convex combination of rewards.
Revised: We propose a correction term – which we call Divergence Correction (DC)– based on the R ´ enyi divergence between policies which allows us, in principle, to recover the optimal policy for transfer for any convex combination of rewards.

Original: We demonstrate a practical implementation of these methods in continuous action spaces using adaptive importance sampling and compare the approaches introduced here: max-ent GPI and DC with optimism(Haarnoja et al., 2018a) and Conditional Q functions (Schaul et al., 2015) in a variety of non-trivial continuous action transfer tasks.
Revised: We demonstrate a practical implementation of these methods in continuous action spaces using adaptive importance sampling and compare the approaches introduced here: max-ent GPI and DC with optimism(Haarnoja et al., 2018a) and Conditional Q functions (Schaul et al., 2015) in a variety of non-trivial continuous action transfer tasks.

Original:  B ACKGROUND 2. M ULTI - TASK RL
Revised:  B ACKGROUND 2. M ULTI - TASK RL

Original: We consider Markov Decision Processes deﬁned by the tuple M containing: a state space S , action space A , a start state distribution p ( s 1 ) , a transition function p ( s t +1 | s t , a t ) , a discount γ ∈ [0 , 1) and a reward function r ( s t , a t , s t +1 ) .
Revised: We consider Markov Decision Processes deﬁned by the tuple M containing: a state space S , action space A , a start state distribution p ( s 1 ) , a transition function p ( s t +1 | s t , a t ) , a discount γ ∈ [0 , 1) and a reward function r ( s t , a t , s t +1 ) .

Original: The objective of RL is to ﬁnd a policy π ( a | s ) :
Revised: The objective of RL is to ﬁnd a policy π ( a | s ) :

Original: S → P ( A ) which maximises the discounted expected return from any state J ( π ) =
Revised: S → P ( A ) which maximises the discounted expected return from any state J ( π ) =

Original: E π, M [ (cid:80) ∞ τ = t γ τ − t r τ ] where the expected reward is dependent on the policy π and the MDP M .
Revised: E π, M [ (cid:80) ∞ τ = t γ τ − t r τ ] where the expected reward is dependent on the policy π and the MDP M .

Original: We formalize transfer as in Barreto et al.
Revised: We formalize transfer as in Barreto et al.

Original: (2017); Haarnoja et al.
Revised: (2017); Haarnoja et al.

Original: (2018a), as the desire to perform well across all tasks in a set M ∈ T (cid:48)
Revised: (2018a), as the desire to perform well across all tasks in a set M ∈ T (cid:48)

Original: after having learned policies for tasks M ∈ T , without additional experience.
Revised: after having learned policies for tasks M ∈ T , without additional experience.

Original: We assume that T and T (cid:48) are related in two ways: all tasks share the same state transition function, and tasks in T (cid:48) can be expressed as convex combinations of rewards associated with tasks in set T .
Revised: We assume that T and T (cid:48) are related in two ways: all tasks share the same state transition function, and tasks in T (cid:48) can be expressed as convex combinations of rewards associated with tasks in set T .

Original: So if we write the reward functions for tasks in T as the vector φ = ( r 1 , r 2 , . . . ) , tasks in T (cid:48) can be expressed as r w = φ · w .
Revised: So if we write the reward functions for tasks in T as the vector φ = ( r 1 , r 2 , . . . ) , tasks in T (cid:48) can be expressed as r w = φ · w .

Original: We focus on combinations of two policies r b
Revised: We focus on combinations of two policies r b

Original: =
Revised: =

Original: br i
Revised: br i

Original: + (1 − b ) r j
Revised: + (1 − b ) r j

Original: but the methods can be extended to more than two tasks.
Revised: but the methods can be extended to more than two tasks.

Revised: We refer to a transfer method as optimal, if it achieves optimal returns on tasks in T (cid:48) , only experience on tasks T .

Original: 2. S UCCESSOR F EATURES
Revised: 2. S UCCESSOR F EATURES

Original: Successor Features (SF) (Dayan, 1993) and Generalised Policy Improvement (GPI) (Barreto et al., 2017; 2018) provide a principled solution tothe transfer in the setting deﬁned above.
Revised: Successor Features (SF) (Dayan, 1993) and Generalised Policy Improvement (GPI) (Barreto et al., 2017; 2018) provide a principled solution to transfer in the setting deﬁned above.

Original: SF make the additional assumption that the reward feature φ is fully observable, that is, the agent has access to the rewards of all tasks in T but not T (cid:48) during training on each individual task.
Revised: SF make the additional assumption that the reward feature φ is fully observable, that is, the agent has access to the rewards of all tasks in T but not T (cid:48) during training on each individual task.

Original: The key observation of SF representations is that linearity of the reward r w with respect to the features φ implies the following decomposition of the value policy of π as
Revised: The key observation of SF representations is that linearity of the reward r w with respect to the features φ implies the following decomposition of the value policy of π :

Original: [Equation]
Revised: [Equation]

Original: where ψ π is the expected discounted sum of features φ induced by policy π .
Revised: where ψ π is the expected discounted sum of features φ induced by policy π .

Original: This decomposition allows us to compute the action-value for π on any task w by learning ψ π .
Revised: This decomposition allows us to compute the action-value for π on any task w by learning ψ π .

Original: If we have a set of policies π 1 , π 2 , ..., π n indexed by i ,  GPI provides a principled approach to transfer on task π w .
Revised: If we have a set of policies π 1 , π 2 , ..., π n indexed by i , SF and GPI provide a principled approach to transfer on task π w .

Original: Namely, GPI guarantees that the  GPI policy π GPI w [Equation] t ))
Revised: Namely, we act according to the deterministic GPI policy π GPI w ( s t ) ≡ arg max a t Q GPI w ( s t , a t ))

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original:  has a return at least as good as any component policy, that is, V π 
Revised: The GPI theorem guarantees the GPI policy has a return at least as good as any component policy, that is, V π GPI w

Original: [Equation]
Revised: [Equation]

Original: 2. M AXIMUM E NTROPY RL
Revised: 2. M AXIMUM E NTROPY RL

Original: The maximum entropy (max-ent) RL objective augments the rewardwith an entropy term to favor entropic solutions
Revised: The maximum entropy (max-ent) RL objective augments the reward to favor entropic solutions

Original: [Equation]
Revised: [Equation]

Original: where α is a parameter that determines the relative importance of the entropy term.
Revised: where α is a parameter that determines the relative importance of the entropy term.

Original: This objective has been considered in a number of works including Kappen (2005); Todorov (2009);
Revised: This objective has been considered in a number of works including Kappen (2005); Todorov (2009);

Original: Haarnoja et al.
Revised: Haarnoja et al.

Original: (2017; 2018a)
Revised: (2017; 2018a)

Original: ; Ziebart et al.
Revised: ; Ziebart et al.

Original: ; Fox et al.
Revised: ; Fox et al.

Original: We deﬁne the action-value Q π associated with eq. 3 as
Revised: We deﬁne the action-value Q π associated with eq. 3 as

Original: [Equation]
Revised: [Equation]

Original: (notice Q π ( s t , a t ) does not include any entropy terms for the state s t ).
Revised: (notice Q π ( s t , a t ) does not include any entropy terms for the state s t ).

Original: Soft Q iteration
Revised: Soft Q iteration

Original: [Equation]
Revised: [Equation]

Original: where π ( a t | s t ) ∝ exp( 1 α Q ( s t , a t )) converges to the optimal policy with standard assumptions (Haarnoja et al., 2017).
Revised: where π ( a t | s t ) ∝ exp( 1 α Q ( s t , a t )) converges to the optimal policy with standard assumptions (Haarnoja et al., 2017).

Original: The general form of the Boltzmann policy π allows it to capture multimodal solutions.

Original: However, it also makes sampling from π challenging.

Original:  C OMPOSING P OLICIES IN M AX -E NT R EINFORCEMENT L EARNING
Revised:  C OMPOSING P OLICIES IN M AX -E NT R EINFORCEMENT L EARNING

Original: In this section we present two novel approaches for max-ent transfer learning.
Revised: In this section we present two novel approaches for max-ent transfer learning.

Original: In section 4 we then outline a practical method for making use of these results.
Revised: In section 4 we then outline a practical method for making use of these results.

Original: 3. M AX -E NT S UCCESSOR F EATURES AND G ENERALIZED P OLICY I MPROVEMENT
Revised: 3. M AX -E NT S UCCESSOR F EATURES AND G ENERALIZED P OLICY I MPROVEMENT

Original: We introduce max-ent SF, which provide a practical method for computing the value of a maximum entropy policy under any convex combination of rewards.
Revised: We introduce max-ent SF, which provide a practical method for computing the value of a maximum entropy policy under any convex combination of rewards.

Original: We then formulate GPI  (Barreto et al., 2017)  for maximum entropy policies.
Revised: We then show the GPI theorem (Barreto et al., 2017) holds for maximum entropy policies.

Original: We deﬁne the action-dependent SF to include the entropy of the policy, excluding the current state, analogous to the max-entropy deﬁnition of Q π in (4):
Revised: We deﬁne the action-dependent SF to include the entropy of the policy, excluding the current state, analogous to the max-entropy deﬁnition of Q π in (4):

Original: [Equation]
Revised: [Equation]

Original: where 1 is a vector of ones of the same dimensionality as φ and we deﬁne the state-dependent successor features as the expected ψ π in analogy with V π ( s ) :
Revised: where 1 is a vector of ones of the same dimensionality as φ and we deﬁne the state-dependent successor features as the expected ψ π in analogy with V π ( s ) :

Original: [Equation]
Revised: [Equation]

Original: The max-entropy action-value of π for any convex combination of rewards w is then given by
Revised: The max-entropy action-value of π for any convex combination of rewards w is then given by

Original: Q π w ( s, a ) =
Revised: Q π w ( s, a ) =

Original: ψ π ( s, a ) · w .
Revised: ψ π ( s, a ) · w .

Original: Max-ent SF allow us to estimate the action-value of previous policies on a new task.
Revised: Max-ent SF allow us to estimate the action-value of previous policies on a new task.

Original: We show that, as in the deterministic case, there is a principled way to combine these estimates.
Revised: We show that, as in the deterministic case, there is a principled way to combine multiple policies using their action-values on task w .

Original: Theorem 3.1 (Max-Ent Generalized Policy Improvement) Let π 1 , π 2 , ..., π n be n policies with α -max-ent action-value functions Q 1 , Q 2 , ..., Q n and value functions V 1 , V 2 , ..., V n .
Revised: Theorem 3.1 (Max-Ent Generalized Policy Improvement) Let π 1 , π 2 , ..., π n be n policies with α -max-ent action-value functions Q 1 , Q 2 , ..., Q n and value functions V 1 , V 2 , ..., V n .

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: Then,
Revised: Then,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where Q π ( s, a )is the α -max-ent action-value function of π , and V π ( s ) is the max-ent state-value function  of π .
Revised: where Q π ( s, a ) and V π ( s ) are the α -max-ent action-value and value function respectively of π .

Original: Proof: See appendix A.1.
Revised: Proof: See appendix A.1.

Original: Once the agent has learned ψ π i ( s, a ) , the SFs of policies π i  , we deﬁne the max-ent GPI policy for task w  as π GPI w
Revised: In our setup, we learn ψ π i ( s, a ) , the SFs of policies π i for each task in T , we deﬁne the max-ent GPI policy for task w ∈ T (cid:48) as π GPI w

Original: [Equation]
Revised: [Equation]

Revised: ( 1 α max i ψ π i ( s, a ) · w ) .

Original: 3. D IVERGENCE C ORRECTION (DC)
Revised: 3. D IVERGENCE C ORRECTION (

Revised: DC)

Original: Haarnoja et al.
Revised: Haarnoja et al.

Original: (2018a) introduced a simple approach to policy combination by estimating the actionvalue for the joint task r b 
Revised: (2018a) introduced a simple approach to policy composition by estimating the actionvalue for the transfer task r b =

Original: = br i
Revised: br i

Original: + (1 − b ) r j from the optimal action-values of the component tasks [Equation] j
Revised: + (1 − b ) r j from the optimal action-values of the component tasks Q i and Q j

Original: [Equation]
Revised: [Equation]

Original: When using Boltzmann policies deﬁned by Q , the resulting policy, π Optb ( a | s ) ∝ exp [Equation] 1 α Q Optb ( s, a )) , is the product distribution of the two component policies.
Revised: When using Boltzmann policies deﬁned by Q , the resulting policy, π Optb ( a | s ) ∝ exp( 1 α Q Optb ( s, a )) , is the product distribution of the two component policies.

Original: We refer to π Optb as the  “optimistic”  policy, as it acts according to the optimistic assumption that the optimal returns of Q i and Q j will be, simultaneously, achievable.
Revised: We refer to π Optb as the compositionally “optimistic” (CO) policy, as it acts according to the optimistic assumption that the optimal returns of Q i and Q j will be, simultaneously, achievable 1 .

Original: Haarnoja et al. (2018a) focus on the case when b = 12 and prove that the regret of this estimator can be bounded by the discounted sum of R´enyi divergence of the two component policies.
Revised: Both max-ent GPI we presented above, and CO can, in different ways, fail to transfer well in some situations (see ﬁg. 1 for some examples in tabular case).

Original: Intuitively, if the two policies have large areas of agreement in the actions, the composed policy will perform well.
Revised: Neither approach consistently performs optimally during transfer, even if all component terms are known exactly.

Original: As we analyse later, both max-ent GPI we presented above, and the optimistic policy can, in different ways, fail to transfer well in some situations.
Revised: We desire a solution for transfer that, in principle, can perform optimally.

Original: Here we show, at the cost of learning a function conditional on the task combination, it is in principle possible to recover the optimal policy for theconvex transfer tasks, without direct experience on those tasks, by correcting for the optimistic bias in Q Optb .
Revised: Here we show, at the cost of learning a function conditional on the task weightings b , it is in principle possible to recover the optimal policy for the transfer tasks, without direct experience on those tasks, by correcting for the compositional optimism bias in Q Optb .

Original: For simplicity, as in Haarnoja et al.
Revised: For simplicity, as in Haarnoja et al.

Original: (2018a), we restrict this to the case with only 2 tasks, but it can be extended to multiple tasks.
Revised: (2018a), we restrict this to the case with only 2 tasks, but it can be extended to multiple tasks.

Revised: The correction term for CO uses a property noted, but not exploited in Haarnoja et al.

Revised: (2018a).

Revised: The bias in Q Opt is related to the the discounted sum of R´enyi divergences of the two component policies.

Revised: Intuitively, if the two policies result in trajectories with low divergence between the policies in each state, the CO assumption that both policies can achieve good returns is approximately correct.

Revised: When the divergences are large, the CO assumption is being overly optimistic and the correction term will be large.

Original: Theorem 3.2 (DC Optimality) Let π i , π
Revised: Theorem 3.2 (DC Optimality) Let π i , π

Original: j be α max-ent optimal policies for tasks with rewards r i and r j with max-ent action-value functions
Revised: j be α max-ent optimal policies for tasks with rewards r i and r j with max-ent action-value functions

Original: Q i , Q j .
Revised: Q i , Q j .

Original: Deﬁne C ∞ b ( s t , a t ) as the ﬁxed point of
Revised: Deﬁne C ∞ b ( s t , a t ) as the ﬁxed point of

Original: C ( k +1)
Revised: C ( k +1)

Original: b
Revised: b

Original: [Equation]
Revised: [Equation]

Original: Given the conditions for Soft Q convergence, the max-ent optimal Q ∗ b ( s, a ) for r b = br i + (1 − b ) r
Revised: Given the conditions for Soft Q convergence, the max-ent optimal Q ∗ b ( s, a ) for r b = br i + (1 − b ) r

Original: j is
Revised: j is

Original: Q ∗ b ( s, a ) = bQ i ( s, a ) + (1 − b ) Q j ( s, a ) − C ∞ b ( s, a ) ∀ s ∈ S , a ∈ A , b ∈ [0 , 1] .

Original: [Table]
Revised: [Equation]

Original: Proof: See appendix A.2.
Revised: Proof: See appendix A.2.

Original: We call this Divergence Correction (DC) as the quantity C ∞ b is related to the R´enyi divergence between policies (see appendix A.2 for details).
Revised: We call this Divergence Correction (DC) as the quantity C ∞ b is related to the R´enyi divergence between policies (see appendix A.2 for details).

Original: Learning C ∞ b does not require any additional information (in principle) than that required to learn policies π 1 and π 2 .
Revised: Learning C ∞ b does not require any additional information (in principle) than that required to learn policies π i and π j .

Original: Unlike with
Revised: Unlike with

Original: SF, it is not necessary to observe other task features while training the policies.
Revised: SF, it is not necessary to observe other task features while training the policies.

Original: On the other hand, unlike with GPI, which can be used to naturally combine any number of tasks with arbitrary weight vectors w , in order to apply DC one must estimate C ∞ b ( s, a ) for all values of b .
Revised: On the other hand, unlike with GPI, which can be used to naturally combine any number of tasks with arbitrary weight vectors w , in order to apply DC one must estimate C ∞ b ( s, a ) for all values of b .

Original: so the complexity of learning C ∞ increases signiﬁcantly if more than 2 tasks are combined.
Revised: so the complexity of learning C ∞ increases signiﬁcantly if more than 2 tasks are combined.

Original:  Table 1 provides a comparison on the properties of the methods we consider here.
Revised: Supplementary Table 1 provides a comparison on the properties of the methods we consider here.

Original: We also compare with simply learning a conditional Q function Q ( s, a | b ) (CondQ) (e.g. Schaul et al., 2015; Andrychowicz et al., 2017).
Revised: We also compare with simply learning a conditional Q function Q ( s, a | b ) (CondQ) (e.g. Schaul et al., 2015; Andrychowicz et al., 2017).

Original: As with GPI, this requires observing the full set of task features φ , in order to compute r b for arbitrary b .
Revised: As with GPI, this requires observing the full set of task features φ , in order to compute r b for arbitrary b .

Revised: In this section we have introduced two new theoretical approaches to max-ent transfer composition: max-ent GPI and DC.

Revised: We have shown how these are related to relevant prior methods.

Revised: In the next section we address the question of how to practically learn and sample with these approaches in continuous action spaces.

Original: I MPLEMENTATION USING  A DAPTIVE I MPORTANCE S AMPLING 
Revised:  A DAPTIVE I MPORTANCE S AMPLING FOR B OLTZMAN P OLICIES A LGORITHM

Original: Compositional transfer appears to be the most useful in high-dimensional, continuous actionspaces, since in such action spacesit is often possible to simultaneously work towards achieving multiple objectives.
Revised: The control of robotic systems with high-dimensional continuous action spaces is a promising use case for the ideas presented in this paper. Such control problems may allow for multiple solutions, and can exhibit exploitable compositional structure.

Original: Sampling the Boltzmann policies deﬁned by Q in continuous action spaces is challenging, particularly during transferwhen we need to sample from joint distributions of existing policies.
Revised: Unfortunately, learning and sampling of general Boltzmann policies deﬁned over continuous action spaces is challenging. While this can be mitigated by learning a parametric sampling distribution, during transfer we want to sample from the Boltzmann policy associated with a newly synthesized action-value function without having to learn such an approximation ﬁrst.

Original: We now present an approach using adaptive importance sampling with mixture proposal distributions to learn and transfer with max-ent policies.
Revised: To address this issue we introduce Adaptive Importance Sampling for Boltzmann Policies (AISBP), a method which provides a practical solution to this challenge. In the following we parametrise all functions with neural nets (denoting parameters by the subscript θ ), including the soft action-value for reward i : Q iθ Q ( s, a ) ; the associated soft value function V iθ V ( s ) and a proposal distribution q iθ q ( a | s ) , the role of which we explain below. We use an off-policy algorithm, so that experience generated by training on policy i can be used to improve policy j .

Original: Our approach is motivated by two observations: (i) batch computation in modern GPUs and CPUs makes importance sampling with a relatively large number of samples efﬁcient (since the samples can be treated as a batch and computed efﬁciently), (ii) the product distribution for a mixture of normal distributions can be tractably sampled from.
Revised: This is especially important since our analysis requires the action-value Q i ( s, a ) to be known in all states. This is less likely to be the case for a on on-policy algorithm, that only updates Q i using trajectories generated by policy π i . During training experience generated by all tasks are stored in a replay buffer R , and mini-batches are sampled uniformly and used to update all function approximators.

Revised: Soft Q iteration (see eq. 4) is used to learn Q i and V i .

Original: This motivates us to use adaptive importance sampling with a mixture of (truncated) normal distributions for the proposal distribution.
Revised: These updates are, in principle, straightforward using transitions sampled from the replay buffer. Sampling from the Boltzmann policy deﬁned by Q iθ Q , π i ( a | s ) ∝ exp 1 α Q iθ Q ( s, a ) is challenging as is estimating the partition function (the log of which is also the value, c.f. Eq. 6). One approach is to ﬁt an expressible, tractable sampler, such as a stochastic neural network to approximate π i (e.g.

Original: 4. L EARNING M AX -E NT B OLTZMANN POLICIES WITH A DAPTIVE I MPORTANCE S AMPLING
Revised: Haarnoja et al., 2018a). This approach works well when learning a single policy.

Original: The basic method we use is to learn proposal distributions for each policy and use this to importance  sample  the policy for estimating the other terms.
Revised: However, during transfer this may require learning a new sampler for each new value composition. AISBP instead uses importance sampling to sample π and estimate the partition function. The scalability of this approach is improved by using using a learned proposal distribution q θ q ( a | s ) , and by observing that modern architectures allow for efﬁcient batch computation of a large number of importance samples. To facilitate transfer we restrict the parametric form of the proposals to mixtures of (truncated) Normal distributions.

Original: Speciﬁcally, for each policy  we learn an actionvalue Q θ Q ( s, a ) , and value V θ V ( s ) network, and a proposal distribution q  θ q ( a | s ) .
Revised: The well-known result that the product of Normal distributions can be computed in closed-form then allows us to construct effective compositional proposals during transfer. More formally, for each policy in T we learn an action-value Q iθ Q ( s, a ) , and value V iθ V ( s ) network, and a proposal distribution q i θ q ( a | s ) (we drop the task index i here when writing the losses for nota- tional clarify, and write the losses for a single policy). The proposal distribution is a mixture of M truncated Normal distributions N T , truncated to the square a ∈ [ − 1 , 1) n with diagonal covariances

Original:  The proposal distribution is learned by minimizing the forward KL divergence with the Boltzmann policydistribution π ( a | s ) ∝ exp 1 α Q θ Q
Revised: [Equation] The proposal distribution is optimized by minimizing the forward KL divergence with the Boltzmann policy π ( a | s ) ∝ exp 1 α Q θ Q

Original: ( s, a ) .
Revised: ( s, a ) .

Original: We use the forward divergence because it is “zero avoiding” and will typically over-estimate the support of π (Murphy, 2012) which is desirablewhen for the use as proposal distribution. The loss on the proposal distribution is then
Revised: This KL is “zero avoiding” and over-estimates the support of π (Murphy, 2012) which is desirable for a proposal distribution (Gu et al., 2015),

Original: [Equation]
Revised: [Equation]

Original: (12) where the expectation is over some off-policy state density ρ β generated by exploration policy β .
Revised: where the expectation is over the replay buffer state density.

Original: However, this proposal loss itself requires we sample from π , so we importance sample.
Revised: The inner expectation in the proposal loss itself requires sampling from π .

Original: We use self-normalized importance weighting so that we do not need to compute the partition function for π . An obvious choice for the proposal distribution would be q θ q ( a | s ) , however, we found this to be unstable so we use a target proposal distribution q θ (cid:48)
Revised: We approximate his expectation by self-normalized importance sampling and use a target proposal distribution p ( a t

Original: q and use a mixture distribution  of the target proposal distributions for all policies.
Revised: | s t ) which is a mixture distribution consisting of the proposals for all policies along with a uniform distribution.

Original: The estimator for all losses are given in Appendix B.
Revised: For batchsize B and N proposal samples the estimator of the proposal loss is then L

Original: With the learned proposal distribution we can then estimate thelosses for Soft Q updates of the action-value and value.
Revised: [Equation] The value function loss is deﬁned as the L2 error on the Soft Q estimate of value

Original: [Equation]
Revised: [Equation]

Original: The target value α log (cid:82) A exp( 1 α Q ( s, a ))

Original: da is estimated usingself-normalized importance sampling with the proposal distribution q θ q ( a | s ) .
Revised: which is estimated using importance sampling to compute the integral. [Equation]

Original: This introduces two sources of bias, due to the selfnormalization and because we apply a concave function log to the expectation.
Revised: This introduces bias due to the ﬁnite-sample approximation of the expectation inside the (concave) log .

Original: In practise, we found this estimator sufﬁcient provided the proposal distribution was closeenough to π .
Revised: In practice we found this estimator sufﬁciently accurate, provided the proposal distribution was close to π .

Original: For acting we usethe the learned proposal distribution to importance  sample the policy.
Revised: We also use importance sampling to sample from π while acting.

Original: We use two common tricks for
Revised: The action-value loss is just the L2 norm with the Soft Q target: [Equation]

Original: DeepRL to make our algorithm more stable: target networks for  V(cid:48) θ V  and  q(cid:48) θ  q  (Mnih et al., 2015; Lillicrap et al., 2015)and parameterizing Q with an advantage function Q θ Q ( s, a )
Revised: To improve stability we employ target networks for the value V θ V (cid:48) and proposal q θ (cid:48) q networks (Mnih et al., 2015; Lillicrap et al., 2015)

Original:  =
Revised: We also parameterize Q as an advantage Q θ Q ( s, a ) =

Original: V θ V ( s )+ A θ A ( s, a ) (Baird, 1994; Wang et al., 2015; Harmon et al., 1995) which is more stable when the advantage is small compared with the value.
Revised: V θ V ( s ) + A θ A ( s, a ) (Baird, 1994; Wang et al., 2015; Harmon et al., 1995) which is more stable when the advantage is small compared with the value.

Original: All of these updates are off-policy, and we learned using a replay buffer of experience.

Original: This algorithm can be straightforwardly extended to learn more than one policy by using estimators for each policy π i . Because our estimators are off-policy, we can store all experience in a single replay buffer, and make use of all experience when training our policies. We implemented this in a distributed framework with actors and learner working in parallel, see supplementary algorithm 1.
Revised: The full algorithm is give in Algorithm Box 1 and more details are provided in appendix C.

Original: 4. I MPORTANCE S AMPLED M AX -E NT GPI
Revised: 4. I MPORTANCE S AMPLED M AX -E NT GPI

Original: We extended our importance sampled approach  to estimate max-ent SF.
Revised: The same importance sampling approach can also be used to estimate max-ent SF.

Original: Max-ent GPI requires us to learn, for each policy
Revised: Max-ent GPI requires us to learn the expected (maximum entropy) features ψ i for each policy

Original: π i ,the expected (maximum entropy) features ψ i , in order to estimate the (entropic) valueof each policy under a new convex combination task w .
Revised: π i , in order to estimate its (entropic) value under a new convex combination task w .

Original: This requires thatour experience tuple  contain the full feature vector φ , rather than just the reward for the policy which generated the experience r i .
Revised: This requires that experience tuple in the replay contain the full feature vector φ , rather than just the reward for the policy which generated the experience r i .

Original: We learn the base policies for each reward r i as in algorithm 1. For learning the max-ent successor features, as with the action-value estimate, we learn astate-dependent target  Υ iθ V along with ψ iθ Q .
Revised: Given this information ψ θ ψ and Υ θ Υ can be learned with analogous updates to V and Q , which again requires importance sampling to estimate Υ . As with V θ V , we use a target network for Υ θ (cid:48) Υ and advantage parametrization.

Original: [Equation]
Revised: We found that, because these updates when using experience shared between tasks is far off-policy, it is necessary to have a longer target update period than for V .

Original: To learn these quantities we use an analogous approach to the action-values (using the relationship in eq (8) and the proposal distribution q θ q( a | s ) , full details in appendix B).
Revised: Full details are of the losses and samplers are in appendix C. Initialize proposal network θ q , value network parameters θ V and action-value network parameters θ Q and replay R while training do (cid:46) in parallel on each actor Obtain parameters θ from learner

Original: We use a target network for Υ θ (cid:48) Υ and found that, because these updates when using experience shared between tasks is far off-policy, it is necessary to have a longer target update period than for V .
Revised: Sample task i ∼ T Roll out episode using q iθ q to importance sample π i ( a | s )

Original: We parametrized ψ as a “psi-vantage” ψ ( s, a ) =
Revised: ∝ exp 1 α Q iθ Q ( s, a ) Add experience to replay R end while while training do (cid:46) in parallel on the learner Sample SARS tuple from R

Original: Υ ( s ) + ψ A ( s, a ) .
Revised: Improve L ( θ q ) , L ( θ V ) , L ( θ Q ) Improve additional losses for transfer L ( θ Υ ) , L ( θ ψ ) , L ( θ C ) , L ( θ V b ) L ( θ Q b ) , if target update period then Update target network parameters θ V (cid:48)

Revised: ← θ V , θ q (cid:48)

Revised: ← θ q , θ Υ (cid:48) ← θ Υ , θ V (cid:48)

Revised: b ← θ V b end if end while

Original: 4. D IVERGENCE C ORRECTION
Revised: 4. D IVERGENCE C ORRECTION

Original: All that is required for transfer using optimistic policy combination (eq. 11, Haarnoja et al.
Revised: All that is required for transfer using compositional optimism (eq. 11, Haarnoja et al.

Original: (2018a)) is the max-ent action values of each task.
Revised: (2018a)) is the max-ent action values of each task, so no additional training is required beyond the base policies.

Original: We showed in section 3.2  that if we can learn the ﬁxed point of C ∞ b ( s, a ) we can correct this  optimism and recover the optimal action-value Q ∗ b ( s, a ) .
Revised: In section 3.2 we have shown that if we can learn the ﬁxed point of C ∞ b ( s, a ) we can correct this compositional optimism and recover the optimal action-value Q ∗ b ( s, a ) .

Original: We use the recursive relationship in C ∞ b ( s, a ) to estimate it with a TD-0 estimator.
Revised: We exploit the recursive relationship in C ∞ b ( s, a ) to ﬁt a neural net C θ C ( s, a, b ) with a TD(0) estimator.

Original: However, this requires learning a conditional estimator for any value of b , so as to support arbitrary task combinations.
Revised: This requires learning a conditional estimator for any value of b , so as to support arbitrary task combinations.

Original: Fortunately, since C ∞ b depends only on the policies and transition function, it is possible to estimate C ∞ b for different values of bwithout directly generating roll-outs of this value by sampling b at each update. 
Revised: Fortunately, since C ∞ b depends only on the policies and transition function it is possible to learn an estimator C ∞ b for different values of b by sampling b during each update. As before, we use target networks and an advantage parametrization for C θ C

Original:  We learn C ∞ b as C θ C ( s, a, b ) , for each pair of policies π i , π j with loss[Equation]
Revised: ( s, a, b ) We learn C ∞ b as C θ C ( s, a, b ) , for each pair of policies π i , π j resulting in the loss

Original: [Equation]
Revised: [Equation]

Original: We approximate this losswith an importance sample using a mixture distribution p ( a (cid:48) | s (cid:48) ) of q i , q i and the product distribution q prod ∝ q i ( a (cid:48) | s (cid:48) ) q j ( a (cid:48) | s (cid:48) ) and sample b ∈ U (0 , 1) at each update (the full estimator is given in appendix B). As before with functions we use target networks and an advantage parametrization C ( s, a, b ) = C A ( s, a, b ) + C B
Revised: As with other integrals of the action space, we approximate this loss using importance sampling to estimate the integral.

Original: ( s, b ) . Note that, unlike GPI and CondQ (next section), learning C ∞ b does not require observer φ .
Revised: Note that, unlike GPI and CondQ (next section), learning C ∞ b does not require observing φ while training.

Original: We also considered a heuristic approach where we learned C only for b = 12 (this is typically approximately the largest divergence).
Revised: We also considered a heuristic approach where we learned C only for b = 12 (this is typically approximately the largest divergence).

Original: This avoids the complexity of a conditional estimator and we estimate
Revised: This avoids the complexity of a conditional estimator and we estimate

Original: C ∞ b as ˆ C ∞ b ( s, a ) ∼ 4 b (1 − b ) C ∞ 1 / 2 ( s, a ) .
Revised: C ∞ b as ˆ C ∞ b ( s, a ) ∼ 4 b (1 − b ) C ∞ 1 / 2 ( s, a ) .

Original: This heuristic can be justiﬁed by considering Gaussian policies with similar variance (see appendix C) and often worked well in practise.
Revised: This heuristic, we denote DC-Cheap, can be motivated by considering Gaussian policies with similar variance (see appendix D)

Original: We call this heuristic DC-Cheap, and we can also make use of the max-ent GPI information to correct for over-estimates of  C ∞ b , Q DC − Cheap + GPI ( s, a )
Revised: The max-ent GPI bound can be used to correct for over-estimates of the heuristic C ∞ b , Q DC − Cheap + GPI ( s, a )

Original: = max( Q OPT ( s, a ) − ˆ C ∞ b ( s, a ) , Q GPI ( s, a )) .
Revised: = max( Q OPT ( s, a ) − ˆ C ∞ b ( s, a ) , Q GPI ( s, a )) .

Original: 4. C OND Q
Revised: 4. C OND Q

Original: As a baseline, weconsider instead of learning C ( s, a, b ) , just directly learning a conditional Q function using a similar approach  of sampling b each update Q ( s, a, b ) (Schaul et al., 2015).
Revised: As a baseline, we directly learn a conditional Q function using a similar approach to DC of sampling b each update Q ( s, a, b ) (Schaul et al., 2015).

Original: This, like GPI but unlike DC, requires observing φ during training so the reward on task b can be estimated.
Revised: This, like GPI but unlike DC, requires observing φ during training so the reward on task b can be estimated.

Original: We provide the full detailsof this in appendix B.
Revised: We provide the full details in appendix C.

Original: 4. S AMPLING C OMPOSITIONAL P OLICIES
Revised: 4. S AMPLING C OMPOSITIONAL P OLICIES

Original: Above we have deﬁned different approaches to estimating the action-value Q b ( s, a ) for the transfer task r b from the existing policies. Sampling from the Boltzmann policy deﬁned by  Q b  is non-trivial. We wish to avoid ofﬂine training of a sampler, as required in Haarnoja et al.
Revised: During transfer we would like to be able to sample from the Boltzmann policy deﬁned by our estimate of the transfer action-value Q b (the estimate is computed using the methods we enumerated above) without having to, ofﬂine, learn a new proposal or sampling distribution ﬁrst (which is the approach employed by Haarnoja et al.

Original: (2018a) during transfer.
Revised: (2018a)).

Original: We take advantage of the productdistribution of proposals being approximately tractable. So we can sample from q ij ( a | s ) ∝  qi ( a | s ) q j ( a | s ) .
Revised: As outlined earlier, we chose the proposal distributions so that the product of proposals is tractable, meaning we can sample from q ijb ( a | s ) ∝ ( q iθ q ( a | s )) b ( q jθ ( a | s ))

Original: If the proposal distributions ﬁt the policies well, the product distribution will be a good proposal distributionfor when the optimistic assumption is valid.{+low, since Q Optb deﬁnes a Boltzmann policy which is the product of the base policies 
+}
Revised: (1 − b ) . This is a good proposal distribution when the CO bias is {+low, since Q Optb deﬁnes a Boltzmann policy which is the product of the base policies 
+}

Original: However, when C ∞ b ( s, a ) is large, then the optimal action during transfer may  be unlikely under the base policies and this  proposal will be poor.
Revised: However, when C ∞ b ( s, a ) is large, meaning the CO bias is large, q ij may not be a good proposal, as we show in the experiments. In this case none of the existing proposal distributions may be a good ﬁt.

Original: To deal with this case, we take advantage of the efﬁciency of computing a large importance sample in the same state space (since it can be efﬁciently batched) and we sample from a mixture distribution of all policies, all policy products and the uniform distribution. 
Revised: Therefore we sample from a mixture distribution of all policies, all policy products and the uniform distribution. [Equation] [Equation] where V A is the volume of the action space.

Original: Empirically, we ﬁnd this is sufﬁcient to approximately recover the optimal policy in our experiments.
Revised: Empirically, we ﬁnd this is sufﬁcient to result in good performance during transfer.

Original: We use the same proposal distribution forall transfer approaches. Alternatively, we could iteratively adapt the proposal distribution online (Kalashnikov et al. (2018) did this for deterministic policies).
Revised: The algorithm for transfer is given in supplementary algorithm 2.

Original:  E XPERIMENTS 5. D ISCRETE , TABULAR ENVIRONMENT
Revised:  E XPERIMENTS 5. D ISCRETE , TABULAR ENVIRONMENT

Original: We ﬁrst consider some illustrative tabular cases of compositional transfer.
Revised: We ﬁrst consider some illustrative tabular cases of compositional transfer.

Original: These highlight where GPI and Optimistic transfer can have high regret (ﬁgure 1).
Revised: These highlight situations in which GPI and CO transfer can perform poorly (Figure 1).

Original: As expected, we ﬁnd that GPI performs well when the optimal  policy is close to one of the existing policies; the optimistic composition performs well when both subtask policies are compatible.
Revised: As expected, we ﬁnd that GPI performs well when the optimal transfer policy is close to one of the existing policies; CO performs well when both subtask policies are compatible.

Original: The“tricky” task  is illustrative of  the sorts of task we will focus on in the next section: It provides non-overlapping rewards for each task  in one corner, while lower value overlapping rewards are provided in the other cornerof the grid world (cf.  1).
Revised: The task we refer to as “tricky” is illustrative of in which the optimal policy for the transfer task does not resemble either existing policy: In the grid world nonoverlapping rewards for each task are provided in one corner of the grid world, while lower value overlapping rewards are provided in the other corner (cf. Fig. 1).

Original: This means that the optimal policy for the joint task does not resemble either existing policy and both GPI andoptimism perform poorly.
Revised: As a consequence both GPI and

Original:  DC performs well in all cases.
Revised: CO perform poorly while DC performs well in all cases.

Original: 5. C ONTINUOUS ACTION SPACES
Revised: 5. C ONTINUOUS ACTION SPACES

Original: We next compare the different approaches in more challenging continuous control tasks.
Revised: We next compare the different approaches in more challenging continuous control tasks.

Original: We train max-ent policies to solve individual tasks using the importance sampling approach from sectionand then assess transfer on convex combinations of the rewards.
Revised: We train max-ent policies to solve individual tasks using the importance sampling approach from section 4 and then assess transfer on convex combinations of the rewards.

Original: We train all approaches using the same experience and proposal distribution.
Revised: All approaches use the same experience and proposal distribution.

Original: [-[Figure] (c) (T)ricky0. 1.0 b10 -7 1010 -3 
 L o g l o ss f r o m o p t i m-] a l (d) LR Regret 0.
Revised: [-[Figure] (c) (T)ricky0. 1.0 b10 -7 1010 -3 
 L o g l o ss f r o m o p t i m-]Figure 2 examines the transfer policies in detail in a simple point-mass task and shows how the estimated C ∞ b corrects the CO Q Opt and dramatically changes the policy.

Original: [-1.0 b10 -7 1010 -3 
-]
Revised: [-1.0 b10 -7 1010 -3 
-]

Original: L o g l o ss f r o m o p t i m a l (e) LU Regret 0.

Original: [-1.0 b10 -7 1010 -3 
-]
Revised: [-1.0 b10 -7 1010 -3 
-]

Original: [-L o g l o ss f r o m o p t i m a l OptimisticGPILTD (f) T Regret 
 V a l u e (g) Opt LR 92.8 93.6 94. V-] a [-l u e (h) GPI LU 48 5157 6066 
-]
Revised: [-L o g l o ss f r o m o p t i m a l OptimisticGPILTD (f) T Regret 
 V a l u e (g) Opt LR 92.8 93.6 94. V-]We then examine conceptually similar tasks in more difﬁcult domains: a [-l u e (h) GPI LU 48 5157 6066 
-]5 DOF planar manipulator reaching task (ﬁgure 3), 3 DOF jumping ball and 8 DOF ant (ﬁgure 4).

Original: [-V a l u e (i) GPI T 48 5157 6066 
 V a l u e (j)-] DC T P o li c y D i v e r g e n c e (k) D ( d ) GPI performs well when the subtasks are incompatible, meaning the optimal policy is near one of the component policies.
Revised: [-V a l u e (i) GPI T 48 5157 6066 
 V a l u e (j)-]We see that DC recovers a qualitatively better policy in all cases.

Original: g ) The optimistic policy performs poorly in these situations, resulting in indecision about which subtask to
Revised: The performance of GPI depends noticeably on the choice of α .

Original: commit to. ( e ) Conversely, when the subpolicies are similar, such as on the LU task, optimistic transfers well while the GPI policy ( h ) does not consistently take advantage of the compatibility of the two tasks to simultaneously achieve both subgoals. ( i ) Neither GPI nor the optimistic policies ( j shows GPI, by the Optimistic policy is similar) perform well when the optimal policy isdissimilar to either existing task policy. The two tricky task policies are compatible in many states but have a high-divergence in the bottom-left corner since the rewards are non-overlapping there ( k ), thus the optimal policy on the composed task is to move to the top right corner where there are overlapping rewards.
Revised: DC-Cheap, which is a simpler heuristic, performs almost as well as DC in the tasks we consider except for the point mass task.

Original: By learning, and correcting for, this future divergence between policies, DC results in optimal policies for all task combinations including tricky ( k ).
Revised: When bounded by GPI (DC-Cheap+GPI) it performs well for the point mass task as well, suggesting simple approximations of C ∞ b may be sufﬁcient in some cases.

Original: We focus onthe “tricky” tasks with multi-modal solutions. Figure 2 examines the transfer policies in detail in a simple point-mass task and shows how the estimated C ∞ b corrects the optimistic Q and dramatically changes the policy. We then examine conceptually similar tasks in more difﬁcult domains: a 5 DOF planar manipulator reaching task (ﬁgure 3), 3 DOF jumping ball and 8 DOF ant (ﬁgure 4).
Revised: We focussed on “tricky” tasks as they are challenging form of transfer.

Original: We see that DC recovers a near optimal policy in all case. The performance of GPI depends noticeably on the choice of α .
Revised: In general, we would expect DC to perform well in most situations where OC performs well, since in this case the correction term

Original: DC-Cheap, which is  asimpler heuristic, performs almost as well as DC in the tasks we consider except for the point mass task. When bounded by GPI (DC-Cheap+GPI) it performs well for the point mass task as well, suggesting simple approximations of C ∞ b may be sufﬁcient in some cases.
Revised: C ∞ b that DC must learn is inconsequential (OC is equivalent to assuming C ∞ b = 0 ). Supplementary ﬁgure 5 demonstrates on a task with non-composible solutions (i.e. C ∞ b is large and potentially challenging to learn), DC continues to perform as well as GPI, slightly better than CondQ, and as expected, OC performs poorly.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have presented two approaches to transfer learning via convex combinations of rewards in the maximum entropy framework: max-ent GPI and DC. 
Revised: We have presented two approaches to transfer learning via convex combinations of rewards in the maximum entropy framework: max-ent GPI and DC. We have shown that, under standard assump-

Original: We have shown that, under standard assumptions, the max-ent GPIpolicy performsat least as well as its component policies, and that DC recovers the optimal policy.
Revised: [Equation] DCDC (a) (L)eft task (b) (T)tricky task(c) (T)tricky task(d) LR regret (e) LU regret (f) T regret (g) Opt LR (h) GPI LU (i) GPI T (j) DC T (k) D 1 / 2 D 1 / COGPIDC ( d ) GPI performs well when the subtasks are incompatible, meaning the optimal policy is near one of the component policies.

Original: Todorov (2009) and (Saxe et al., 2017; van Niekerk et al., 2018) previously considered optimal composition of max-ent policies.
Revised: ( g ) CO performs poorly in these situations, resulting in indecision about which subtask to commit to. ( e ) Conversely, when the subpolicies are compatible, such as on the LU task, CO transfers well while the GPI policy ( h ) does not consistently take advantage of the compatibility of the two tasks to simultaneously achieve both subgoals. ( f ) Neither GPI nor CO policies ( i shows the GPI policy, but CO is similar) perform well when the optimal transfer policy is dissimilar to either existing task policy.

Original: However, these approaches require stronger assumptions than max-ent SF or DC, namely that reward states  are absorbing and that the 
Revised: The two tricky task policies are compatible in many states but have a high-divergence in the bottom-left corner since the rewards are non-overlapping there ( k ), thus the optimal policy on the composed task is to move to the top right corner where there are overlapping rewards. By learning, and correcting for, this future divergence between policies, DC results in optimal policies for all task combinations including tricky ( j ).

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: the max-ent GPI policy performs at least as well as its component policies recovers the optimal transfer policy.

Original: [Figure] [Figure] R e t u r n
Revised: Todorov (2009) and (Saxe et al., 2017; van Niekerk et al., 2018) previously considered optimal composition of max-ent policies.

Original: OptimisticGPIDCDC
Revised: However, these approaches

Original: [Figure]
Revised: [Figure]

Original: GPI

Original: [Figure]
Revised: [Figure]

Original: xx0. (c) Daa  (d) Q Opt aa  (e) C ∞ 1 /aa (f) Q LTD
Revised: and DC consistently go to the blue square (optimal), CondQ and CO do not.

Original: [Figure] joint reward is restricted to the softmax of the component rewards (soft OR).
Revised: require stronger assumptions than max-ent SF or DC, namely that reward states are absorbing and that the joint reward is restricted to the softmax of the component rewards (soft OR).

Original: By contrast, DC does not restrict the class of MDPs and learns how compatible policies are, allowing approximate recovery of optimal transfer policies both when the component rewards are jointly achievable (AND), and when only one sub-goal can be achieved (OR).
Revised: By contrast, DC does not restrict the class of MDPs and learns how compatible policies are, allowing approximate recovery of optimal transfer policies both when the component rewards are jointly achievable (AND), and when only one sub-goal can be achieved (OR).

Original: We have compared our methods with conditional action-value functions (CondQ) (Schaul et al., 2015, e.g.) and optimistic policy combination (Haarnoja et al., 2018a).
Revised: We have compared our methods with conditional action-value functions (CondQ) (Schaul et al., 2015, e.g.) and optimistic policy combination (Haarnoja et al., 2018a).

Original: Further, we have presented  a practical algorithm for training DC and max-ent GPI models in continous action spaces using adaptive importance sampling.
Revised: Further, we have presented AISBP, a practical algorithm for training DC and max-ent GPI models in continuous action spaces using adaptive importance sampling.

Original: We have compared these approaches, along with heuristic approximations of DC, and demonstrated that DC recovers an approximately optimal policy during transfer across a variety of high-dimensional control tasks.
Revised: We have compared these approaches, along with heuristic approximations of DC, and demonstrated that DC recovers an approximately optimal policy during transfer across a variety of high-dimensional control tasks.

Original: Empirically we have found CondQ may be harder to learn than DC, and it requires additional observation of φ during training.[Figure] P [Equation]
Revised: Empirically we have found CondQ may be harder to learn than DC, and it requires additional observation of φ during training.

Original: A. M AX -E NT G ENERALIZED P OLICY I MPROVEMENT
Revised: A. M AX -E NT G ENERALIZED P OLICY I MPROVEMENT

Original: Theorem 3.1 (Max-Ent Generalized Policy Improvement) Let π 1 , π 2 , ..., π n be n policies with α -max-ent action-value functions Q 1 , Q 2 , ..., Q n and value functions V 1 , V 2 , ..., V n .
Revised: Theorem 3.1 (Max-Ent Generalized Policy Improvement) Let π 1 , π 2 , ..., π n be n policies with α -max-ent action-value functions Q 1 , Q 2 , ..., Q n and value functions V 1 , V 2 , ..., V n .

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where Q π ( s, a )is the α -max-ent action-value function of π , and V π ( s ) is the max-ent state-value function  of π .
Revised: where Q π ( s, a ) and V π ( s ) are the α -max-ent action-value and value function respectively of π .

Original: For brevity we denote Q max ≡ max
Revised: For brevity we denote Q max ≡ max

Original: i Q i .
Revised: i Q i .

Original: Deﬁne the soft Bellman operator associated with policy π as
Revised: Deﬁne the soft Bellman operator associated with policy π as

Original: [Equation]
Revised: [Equation]

Original: Haarnoja et al.
Revised: Haarnoja et al.

Original: (2018b) have pointed out that the soft Bellman operator T π corresponds to a conventional, “hard”, Bellman operator deﬁned over the same MDP but with reward r π ( s, a, s (cid:48) )
Revised: (2018b) have pointed out that the soft Bellman operator T π corresponds to a conventional, “hard”, Bellman operator deﬁned over the same MDP but with reward r π ( s, a, s (cid:48) )

Original: =
Revised: =

Original: r ( s, a, s (cid:48) ) + γα E p ( s (cid:48) | s,a )
Revised: r ( s, a, s (cid:48) ) + γα E p ( s (cid:48) | s,a )

Original: [ H [ π ( ·| s (cid:48) )]] .
Revised: [ H [ π ( ·| s (cid:48) )]] .

Original: Thus, as long as r ( s, a, s (cid:48) ) and H [ π ( ·| s (cid:48) )] are bounded, T π is a contraction with Q π as its ﬁxed point.
Revised: Thus, as long as r ( s, a, s (cid:48) ) and H [ π ( ·| s (cid:48) )] are bounded, T π is a contraction with Q π as its ﬁxed point.

Original: Appplying T π to Q max ( s, a ) we have:
Revised: Appplying T π to Q max ( s, a ) we have:

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: if we apply T π
Revised: if we apply T π

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: the soft Bellman operator induced by policy π i [Equation] to Q max ( s, a )
Revised: the soft Bellman operator induced by policy π i , to Q max ( s, a )

Original: [Equation]
Revised: [Equation]

Original: we obtain:
Revised: we obtain:

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Revised: i Q max

Revised: [Equation]

Revised: s, a

Revised: [Equation]

Revised: = r

Original:  We now note that the Kullback-Leibler divergence between π i and π can be written as
Revised: [Equation] [Equation] [Equation] We now note that the Kullback-Leibler divergence between π i and π can be written as

Original: [Equation]
Revised: [Equation]

Original: The quantity above, which is always nonnegative, will be useful in the subsequent derivations.
Revised: The quantity above, which is always nonnegative, will be useful in the subsequent derivations.

Original: Next we write
Revised: Next we write

Original: [Equation]
Revised: [Equation]

Original: From (16) we have that
Revised: From (20) we have that

Original: [Equation]
Revised: [Equation]

Original:  all i
Revised: for all i.

Original: [Equation]

Original: Using the contraction and monotonicity of the soft Bellman operator T π we have
Revised: Using the contraction and monotonicity of the soft Bellman operator T π we have

Original: [Equation]
Revised: [Equation]

Original: for all i.
Revised: for all i.

Original: We have just showed (9).
Revised: We have just showed (9).

Original: In order to show (10)
Revised: In order to show (10)

Original: [Equation]
Revised: [Equation]

Original: we note that
Revised: we note that

Original: V

Original: [Equation]
Revised: [Equation]

Revised: | s )]

Revised: + E a ∼ π [ Q π ( s, a )] ≥ αH [ π ( ·| s )]

Revised: + E a ∼ π [ Q max ( s, a )]

Revised: = α log Z π ( s ) .

Original: Similarly, we have, for all i ,
Revised: Similarly, we have, for all i ,

Original: [Equation]
Revised: [Equation]

Original: The bound (10) follows from (17) and (18).
Revised: The bound (10) follows from (21) and (22).

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.2 (DC Optimality) Let π i , π
Revised: Theorem 3.2 (DC Optimality) Let π i , π

Original: j be α max-ent optimal policies for tasks with rewards r i and r j with max-ent action-value functions
Revised: j be α max-ent optimal policies for tasks with rewards r i and r j with max-ent action-value functions

Original: Q i , Q j .
Revised: Q i , Q j .

Original: Deﬁne C ∞ b ( s t , a t ) as the ﬁxed point of
Revised: Deﬁne C ∞ b ( s t , a t ) as the ﬁxed point of

Original: C ( k +1)
Revised: C ( k +1)

Original: b
Revised: b

Original: [Equation]
Revised: [Equation]

Original:  Given the conditions for Soft Q convergence, the max-ent optimal Q ∗ b ( s, a ) for r b = br i + (1 − b ) r
Revised: t [Equation] Given the conditions for Soft Q convergence, the max-ent optimal Q ∗ b ( s, a ) for r b = br i + (1 − b ) r

Original: j is
Revised: j is

Original: Q ∗ b ( s, a ) = bQ i ( s, a ) + (1 − b ) Q j ( s, a ) − C ∞ b ( s, a ) ∀ s ∈ S , a ∈ A , b ∈ [0 , 1] .
Revised: Q ∗ b ( s, a ) = bQ i ( s, a ) + (1 − b ) Q j ( s, a ) − C ∞ b ( s, a ) ∀ s ∈ S , a ∈ A , b ∈ [0 , 1] .

Original: We follow a similar approach to Haarnoja et al.
Revised: We follow a similar approach to Haarnoja et al.

Original: (2018a)
Revised: (2018a)

Original: but without making approximations and generalizing to all convex combinations.
Revised: but without making approximations and generalizing to all convex combinations.

Original: First note that since π i and π j are optimal then π i ( a | s ) =
Revised: First note that since π i and π j are optimal then π i ( a | s ) =

Original: exp( 1 α ( Q i ( s, a ) − V
Revised: exp( 1 α ( Q i ( s, a ) − V

Original: i
Revised: i

Original: ( s ))) .
Revised: ( s ))) .

Original: For brevity we use s and s (cid:48) notation rather than writing the time index.
Revised: For brevity we use s and s (cid:48) notation rather than writing the time index.

Original: Deﬁne
Revised: Deﬁne

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and consider soft Q-iteration on r b starting from Q (0) b .
Revised: and consider soft Q-iteration on r b starting from Q (0) b .

Original: We prove, inductively, that at each iteration
Revised: We prove, inductively, that at each iteration

Original: Q ( k ) b = bQ
Revised: Q ( k +1) b = bQ

Original: i
Revised: i

Revised: [Equation]

Revised: s

Original: [Equation]
Revised: [Equation]

Revised: a

Revised: [Equation]

Original: +
Revised: +

Revised: [Equation]

Revised: [Equation]

Revised: b ) Q

Original: [Equation]
Revised: [Equation]

Revised: C ( k +1)

Revised: ( s, a ) .

Original: This is true by deﬁnition for k = 0 .
Revised: This is true by deﬁnition for k = 0 .

Original: [Equation]
Revised: [Equation]

Original: Since soft Q-iteration converges to the α max-ent optimal soft Q then equation 19 holds at the limit.
Revised: Since soft Q-iteration converges to the α max-ent optimal soft Q then equation 31 holds at the limit.

Original: One can get an intuition for C ∞ b ( s, a ) by noting that
Revised: One can get an intuition for C ∞ b ( s, a ) by noting that

Original: [Equation]
Revised: [Equation]

Original: where D b is the R´enyi divergence of order b .
Revised: where D b is the R´enyi divergence of order b .

Original: C ∞ b ( s, a ) can be seen as the discount sum of divergences, weighted by the unnormalized product distribution π 1 ( a | s )
Revised: C ∞ b ( s, a ) can be seen as the discount sum of divergences, weighted by the unnormalized product distribution π 1 ( a | s )

Original: b π 2 ( a | s ) 1 − b .
Revised: b π 2 ( a | s ) 1 − b .

Revised: A. N POLICIES

Revised: It is possible to extend Theorem 3.2 to the case with N policies in a straightforward way.

Revised: Theorem A.1 (Multi-policy DC Optimality) Let π 1 , π 2 , ..., π N be α max-ent optimal policies for tasks with rewards r 1 , r 2 , ..., r N with max-ent action-value functions Q 1 , Q 2 , ..., Q N .

Revised: Deﬁne C ∞ w ( s t , a t ) as the ﬁxed point of

Revised: C ( k

Original: B A LGORITHM DETAILS B. I MPORTANCE S AMPLED E STIMATORS
Revised: [Equation] [Equation] w [Equation]

Revised: Given the conditions for Soft Q convergence, the max-ent optimal Q ∗ w ( s, a ) for and convex combination of rewards r w = (cid:80) Ni =1 r i w i is

Revised: [Equation]

Revised: Note that w i refers to component i of the vector w i .

Revised: The proof is very similar to the two reward case above.

Revised: Deﬁne

Revised: [Equation]

Revised: [Equation]

Revised: 32) and again consider soft Q-iteration on r w .

Revised: We prove by induction that at each iteration

Revised: Q

Revised: [Equation]

Revised: Again, this is true by deﬁnition for k = 0 .

Revised: Now we consider a step of Soft Q iteration

Revised: Q ( k +1)

Revised: w

Revised: [Equation]

Revised: [Equation]

Revised: Since soft Q-iteration converges to the α max-ent optimal soft Q then Q ∗ w ( s, a ) =

Original: For clarity we enumerate the various importance-sampled estimators used in our approach.
Revised: [Equation] A . Note that, in practise, estimating C ∞ w may be more challenging for larger N . For compositions of many policies, GPI may be more practical.

Revised: B T HEORETICAL PROPERTIES OF THE COMPOSITION METHODS

Revised: [Table]

Original: All of these updates are off-policy.
Revised: 2 AISBP transfer algorithm Load trained parameters θ Q , θ q , θ s fQ , θ C , θ Q b . Accept transfer task parameter b , transfer method ∈ CO, GPI, DC, CondQ. while testing do

Original: We use a replay buffer R and learn by sampling minibatches of SARS tuples of size B , we index over the batch dimension with l and use s (cid:48) l to denote the state following s l , so the tuple consists of ( s l , a l , r l , s(cid:48) l )  . 
Revised: Importance sample transfer policy π b ( a | s ) ∝ exp 1 α Q method ( s, a ) with mixture proposal p b ( a | s ) θ q . end while C. A LL LOSSES AND ESTIMATORS We use neural networks to parametrize all quantities.

Original: For importance sampled estimators we sample
Revised: For each policy we learn an action-value Q θ Q ( s, a ) , value V θ V ( s ) and proposal distribution q θ q ( a | s ) . We use target networks for the proposal distribution q θ (cid:48) q ( a | s ) and value V θ (cid:48) V ( s ) . Here we enumerate all of the losses and their estimators.

Revised: We use temporal difference (TD(0)) learning for all the RL losses, so all losses are valid off-policy.

Revised: We use a replay buffer R and learn by sampling minibatches of SARS tuples of size B , we index over the batch dimension with l and use s (cid:48) l to denote the state following s l , so the tuple consists of ( s l , a l , r l , s (cid:48) l ) .

Original:  N actions for each state s l and use a lk to denote sample k for state l . 
Revised: For importance sampled estimators we sample N actions for each state s l and use a lk to denote sample k for state l . We learn a set of n policies, one for each task in T indexed by i . However, we write the losses for a single policy and drop i for notational simplicity. C.2. P ROPOSAL LOSS

Revised: The proposal loss minimizes the KL divergence between the Boltzmann distribution π ( a | s ) ∝ exp( 12 Q ( s, a )) and the proposal distribution.

Revised: [Equation]

Original: Proposal loss The proposal lossequation 12 is sampled using  a mixture distribution p ( a | s ) as described in the main text.
Revised: [Equation] As described in the text, this loss is estimated using importance sampling with a mixture distribution p ( a | s ) containing equally weighted components consisting of the target proposal distribution q θ (cid:48) q ( a | s ) for all policies and the uniform distribution. [Equation]

Original: [Equation]
Revised: [Equation]

Original: where V A is the volume of the action space (which is always bounded in our case)
Revised: where V A is the volume of the action space (which is always bounded in our case).

Revised: The proposal loss is estimated using self-normalized importance sampling

Original: [Equation]
Revised: [Equation]

Original: Value loss The  value estimator for equation
Revised: The soft value loss is [Equation]

Revised: We estimate this using importance sampling with the proposal distribution q θ q ( a | s ) which is trying to ﬁt the policy π

Original: [Equation]
Revised: [Equation]

Revised: C.2. A CTION - VALUE LOSS

Revised: The TD(0) loss for Q θ Q is

Original: [Equation]
Revised: [Equation]

Revised: This does not require importance sampling to estimate and can be straightforwardly estimated as

Original: Action-value loss The action-value estimator for equation
Revised: [Equation] The action-value is parametrized as an advantage function Q θ Q ( s [Equation] a

Original: [Equation]
Revised: [Equation]

Revised: = V θ

Revised: [Equation]

Revised: V ( s

Revised: [Equation]

Revised: + A

Revised: [Equation]

Revised: A

Revised: [Equation]

Revised: s

Revised: [Equation]

Original: Successor Feature Value Estimator
Revised: a ) . C.2. S TATE D EPENDENT S UCCESSOR F EATURES LOSS To facilitate max-ent GPI we learn successor features for each policy, both state-action dependent features ψ θ ψ ( s, a ) and state-dependent Υ θ Υ ( s ) . As with value

Original: [Equation]
Revised: [Equation]

Revised: we use a target network for the state-dependent features Υ θ (cid:48)

Revised: Υ ( s )

Original: [Equation]
Revised: [Equation]

Original:  We use the importance sampled estimate ofα log Z from eq 35, rather than the value network which may be lagging the true partition function.
Revised: This loss is estimated using self-normalized importance sampling with proposal q θ q [Equation] We use the importance sampled estimate of Z from eq 47, rather than the value network which may be lagging the true partition function.

Original: As with the value estimate, we use self-normalized importance sampling to avoid the  weights depending on α log Z ( s l ) (this introduces a bias, but in practise appears to work well). 
Revised: As with the value estimate, we use self-normalized importance sampling to avoid the importance weights depending on α log Z ( s l ) (this introduces a bias, but in practise appears to work well). C.2. S TATE -A CTION D EPENDENT S UCCESSOR F EATURES LOSS

Original: Successor Features Action Value Loss
Revised: The state-action dependent successor feature loss is

Original: [Equation]
Revised: [Equation]

Original: Successor Feature Action-Value
Revised: for which we use the following estimator

Original: estimator
Revised: [Equation] ψ θ [Equation] is parametrized as a “psi-vantage” ψ

Original: [Equation]
Revised: [Equation]

Revised: C.2. DC CORRECTION

Revised: We learn the divergence correction for each pair of policies π i ( a | s ) , π ( a | s ) .

Revised: As described in the text

Revised: [Equation]

Revised: in order to learn C θ C ( s, a, b ) for all b ∈ [0 , 1] , we sample b .

Revised: We also use a target network

Revised: C θ (cid:48) C

Revised: ( s, a, b ) .

Revised: The loss is then

Revised: [Equation]

Revised: This loss is challenging to estimate, due to the dependence on two policies.

Revised: We importance sample using a mixture of all proposal distributions uniform p ( a | s ) (equation 40).

Revised: We denote the samples of b ∼ U (0 , 1) for each batch entry b l .

Revised: The importance sampled estimator is then

Revised: [Equation]

Revised: [Equation]

Revised: We parametrized C θ C as an advantage function C θ C

Revised: [Equation]

Revised: s, a, b ) =

Revised: C

Revised: Aθ CA

Revised: [Equation]

Revised: s, a, b ) + C Bθ CB

Original: [Equation]
Revised: [Equation]

Revised: s, b ) with an additional loss to constrain this parametrization

Original: [Equation]
Revised: [Equation]

Original:  CondQ We also consider, as a control, learning the action-value function conditional on b directly (Schaul et al., 2015).
Revised: which can be straightforwardly estimated by sampling from q [Equation] [Equation] CondQ We also consider, as a control, learning the action-value function conditional on b directly (Schaul et al., 2015), in a similar way to the DC correction.

Original: We learn both a conditional value V θ V ( s, b ) and Q θ Q ( s, a, b ) , again by sampling b uniformly each update.
Revised: We learn both a conditional value V θ Vb ( s, b ) and Q θ Qb ( s, a, b ) , again by sampling b uniformly each update.

Original: [Equation]
Revised: [Equation]

Original: where computing r b for arbitrary b requires φ to have been observed.
Revised: where computing r b for arbitrary b requires φ to have been observed.

Original: We estimate Cond-Q with the same importance samples as C .
Revised: We estimate Cond-Q with the same importance samples as C from p ( a | s ) and again sample b ∼ U (0 , 1) for each entry in the batch.

Original: We use target networks for V θ (cid:48)
Revised: We use target networks for V θ (cid:48)

Original: V ( s, b ) and parametrize Q θ Q ( s, a, b ) =
Revised: V ( s, b ) and parametrize Q θ Q ( s, a, b ) =

Original: V θ  V ( s, b ) +
Revised: V θ (cid:48) V ( s, b ) +

Original: A θ A ( s, a, b ) .
Revised: A θ A ( s, a, b ) .

Original: Algorithm Initialize proposal network θ q , value network parameters θ V and action-value network parameters θ Q and replay R
Revised: The conditional value estimator is

Original: Copy to target nets θ (cid:48) while training do (cid:46) in parallel on each actor
Revised: [Equation]

Original: Obtain parameters θ from learner
Revised: and action-value estimator is

Original: Roll out episode Add experience to replay R end while while training do (cid:46) in parallel on the learner Sample SARS tuple from R Minimize L ( θ q )
Revised: [Equation]

Original: Minimize L ( θ V ) Minimize L ( θ Q ) if target update period then Update target network parameters θ Q (cid:48) ← θ Q , θ q (cid:48) ← θ q end if end while
Revised: [Equation]

Original: C J USTIFICATION FOR THE DC-C HEAP HEURISTIC
Revised: D J USTIFICATION FOR THE DC-C HEAP HEURISTIC

Original: We wish to estimate C ∞ b ( s, a ) (deﬁned in Theorem 3.2) while avoiding learning a conditional function of b .
Revised: We wish to estimate C ∞ b ( s, a ) (deﬁned in Theorem A.1) while avoiding learning a conditional function of b .

Original: We make two (substantial) assumptions to arrive at this approximation.
Revised: We make two (substantial) assumptions to arrive at this approximation.

Original: Firstly, we assume policies π i ( a | s ) , π j ( a | s ) are Gaussian
Revised: Firstly, we assume policies π i ( a | s ) , π j ( a | s ) are Gaussian

Original: [Equation]
Revised: [Equation]

Original:  and the variance σ ( s ) is the same for both policies given a state (it may vary across states).
Revised: (61) and the variance σ ( s ) is the same for both policies given a state (it may vary across states).

Original: Secondly, we assume C ( k ) b ( s, a )
Revised: Secondly, we assume C ( k ) b ( s, a )

Original: =
Revised: =

Original: C ( k ) b ( s ) is independent of action.
Revised: C ( k ) b ( s ) is independent of action.

Original: This is approximately correct when nearby states have similar R´enyi divergences between policies.
Revised: This is approximately correct when nearby states have similar R ´ enyi divergences between policies.

Original: We make use of a result by Gil et al.
Revised: We make use of a result by Gil et al.

Original: (2013) that states that the R´enyi divergence of order b for two
Revised: (2013) that states that the R´enyi divergence of order b for two

Original: Gaussians of the same variance is
Revised: Gaussians of the same variance is

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst deﬁne
Revised: We ﬁrst deﬁne

Original: [Equation]
Revised: [Equation]

Original: From equation
Revised: From equation

Original: [Equation]
Revised: [Equation]

Original: Given these assumptions we show inductively that C ( k ) b ( s, a ) = 4 b (1 − b ) C ( k ) 1 / 2 ( s, a ) ∀ k, b ∈ [0 , 1] .
Revised: Given these assumptions we show inductively that C ( k ) b ( s, a ) = 4 b (1 − b ) C ( k ) 1 / 2 ( s, a ) ∀ k, b ∈ [0 , 1] .

Original: Since C (0) b ( s, a ) = 0 ∀ b ∈ [0 , 1] , a ∈ A , s ∈ S this is true for k = 0 .
Revised: Since C (0) b ( s, a ) = 0 ∀ b ∈ [0 , 1] , a ∈ A , s ∈ S this is true for k = 0 .

Original: We show it holds inductively
Revised: We show it holds inductively

Original: C ( k +1)

Original: b

Original: [Equation]
Revised: [Equation]

Original: Obviously these assumptions are not justiﬁed.
Revised: Obviously these assumptions are not justiﬁed.

Original: However, note that we estimate the true divergence for C ∞ 1 / 2 , i.e. without any assumptions of Gaussian policies and this heuristic is used to estimate
Revised: However, note that we estimate the true divergence for C ∞ 1 / 2 , i.e. without any assumptions of Gaussian policies and this heuristic is used to estimate

Original: C ∞ b from C ∞ 1 / 2 .
Revised: C ∞ b from C ∞ 1 / 2 .

Original: In practise, we ﬁnd this heuristic works in many situations where the policies have similar variance, particulary when bounded by GPI. 
Revised: In practise, we ﬁnd this heuristic works in many situations where the policies have similar variance, particulary when bounded by GPI. [Figure] c [Figure]

Original: D E XPERIMENT DETAILS
Revised: E A DDITIONAL E XPERIMENT F E XPERIMENT DETAILS

Original: All control tasks were simulated using the MuJoCo physics simulator and constructed using the DM control suite (Tassa et al., 2018) which uses the MuJoCo physics simulator (Todorov et al., 2012).
Revised: All control tasks were simulated using the MuJoCo physics simulator and constructed using the DM control suite (Tassa et al., 2018) which uses the MuJoCo physics simulator (Todorov et al., 2012).

Original: The point mass was velocity controlled, all other tasks were torque controlled.
Revised: The point mass was velocity controlled, all other tasks were torque controlled.

Original: The planar manipulator task was based off the planar manipulator in the DM control suite.
Revised: The planar manipulator task was based off the planar manipulator in the DM control suite.

Original: The reward in all tasks was sparse as described in the main text.
Revised: The reward in all tasks was sparse as described in the main text.

Original: During training for all tasks we start states from the randomly sampled positions and orientations.
Revised: During training for all tasks we start states from the randomly sampled positions and orientations.

Original: For the point mass, jumping ball and ant we evaluated transfer starting from the center (in the walker environments, the starting orientation was randomly sampled during transfer, the point mass does not have an orientation).
Revised: For the point mass, jumping ball and ant we evaluated transfer starting from the center (in the walker environments, the starting orientation was randomly sampled during transfer, the point mass does not have an orientation).

Original: For the planar manipulator transfer was tested from same random distribution as in training.
Revised: For the planar manipulator transfer was tested from same random distribution as in training.

Original: For all tasks were learned inﬁnite time horizon policies.
Revised: For all tasks were learned inﬁnite time horizon policies.

Original: Transfer is made challenging by the need for good exploration.
Revised: Transfer is made challenging by the need for good exploration.

Original: That was not the focus on this work.
Revised: That was not the focus on this work.

Original: We aided exploration in several ways: during training we acted according to a higher-temperature policy α e = 2 α .
Revised: We aided exploration in several ways: during training we acted according to a higher-temperature policy α e = 2 α .

Original: We also sampled actions uniformly in an (cid:15) -greedy fashion with (cid:15)
Revised: We also sampled actions uniformly in an (cid:15) -greedy fashion with (cid:15)

Original: = 0 .
Revised: = 0 .

Original: 1 and added
Revised: 1 and added

Original: Gaussian exploration noise during training.
Revised: Gaussian exploration noise during training.

Original: This was sufﬁcient to explore the state space for most tasks.
Revised: This was sufﬁcient to explore the state space for most tasks.

Original: For the planar manipulator and the jumping ball, we found it necessary to induce behavior tasks by learning tasks for reaching the blue target.
Revised: For the planar manipulator and the jumping ball, we found it necessary to induce behavior tasks by learning tasks for reaching the blue target.

Original: This behavior policy was, of course, only used for experience and not during transfer.
Revised: This behavior policy was, of course, only used for experience and not during transfer.

Original: Below we list the hyper-parameters and networks use for all experiment.
Revised: Below we list the hyper-parameters and networks use for all experiment.

Original: The discount γ and α were the only sensitive parameters that we needed to vary between tasks to adjust for the differing magnitudes of returns and sensitivity of the action space between bodies.
Revised: The discount γ and α were the only sensitive parameters that we needed to vary between tasks to adjust for the differing magnitudes of returns and sensitivity of the action space between bodies.

Original: If α is too small then the policies often only ﬁnd one solution and all transfer approaches behave similarly, while for large α the resulting policies are too stochastic and do not perform well.
Revised: If α is too small then the policies often only ﬁnd one solution and all transfer approaches behave similarly, while for large α the resulting policies are too stochastic and do not perform well.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Υ
Revised: Υ

Original: [Table]
Revised: [Table]

Original: period

Original: [Figure]
Revised: [Figure]

Original: Number
Revised: Number

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Number
Revised: Number

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: The state vector was preprocessed by a linear projection of 3 × its dimension
Revised: The state vector was preprocessed by a linear projection of 3 × its dimension

Original: and then a tanh nonlinearity.
Revised: and then a tanh nonlinearity.

Original: All action-state networks ( Q , ψ , C ) consisted of 3 hidden layers with elu non-linearities (Clevert et al., 2015), with both action and preprocessed state projected by linear layers to be of the same dimensionality and used for input the ﬁrst layer.
Revised: All action-state networks ( Q , ψ , C ) consisted of 3 hidden layers with elu non-linearities (Clevert et al., 2015), with both action and preprocessed state projected by linear layers to be of the same dimensionality and used for input the ﬁrst layer.

Original: All value networks and proposal networks consisted of 2 layers with elu non-linearities.
Revised: All value networks and proposal networks consisted of 2 layers with elu non-linearities.

Original: The number of neurons in each layer was varied between environments, but was kept the same in all networks and layers (we did not sweep over this parameter, but choose a reasonable number based on our prior on the complexity of the task).
Revised: The number of neurons in each layer was varied between environments, but was kept the same in all networks and layers (we did not sweep over this parameter, but choose a reasonable number based on our prior on the complexity of the task).

Original: Below we list the per task hyper-parameters
Revised: Below we list the per task hyper-parameters

Original: [Table]
Revised: [Table]