Original: P RIORS , H IERARCHY , AND I NFORMATION A SYMMETRY FOR S KILL T RANSFER IN R EINFORCEMENT L EARNINGAnonymous authors
Revised: P RIORS , H IERARCHY , AND I NFORMATION A SYMMETRY FOR S KILL T RANSFER IN R EINFORCEMENT L EARNING

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The ability to discover behaviours from past experience and transfer them to new tasks is a hallmark of intelligent agents acting sample-efficiently in the real world.
Revised: The ability to discover behaviours from past experience and transfer them to new tasks is a hallmark of intelligent agents acting sample-efficiently in the real world.

Original: Equipping embodied reinforcement learners with the same ability may be crucial for their successful deployment in robotics.
Revised: Equipping embodied reinforcement learners with the same ability may be crucial for their successful deployment in robotics.

Original: While hierarchical and KL-regularized reinforcement learning individually hold promise here, arguably a hybrid approach could combine their respective benefits.
Revised: While hierarchical and KL-regularized reinforcement learning individually hold promise here, arguably a hybrid approach could combine their respective benefits.

Original: Key to these fields is the use of information asymmetry across architectural modules to bias which skills are learnt.
Revised: Key to these fields is the use of information asymmetry across architectural modules to bias which skills are learnt.

Original: While asymmetric choice has a large influence on transferability, existing methods base their choice primarily on intuition in a domain-independent, potentially suboptimal, manner.
Revised: While asymmetry choice has a large influence on transferability, existing methods base their choice primarily on intuition in a domain-independent, potentially suboptimal, manner.

Original: In this paper, we theoretically and empirically show the crucial expressivity-transferability trade-off of skills across sequential tasks, controlled by information asymmetry.
Revised: In this paper, we theoretically and empirically show the crucial expressivity-transferability trade-off of skills across sequential tasks, controlled by information asymmetry.

Original: Given this insight, we introduceAPES , ‘ A ttentive P riors for E xpressive and Transferable S kills’, a hierarchical KL-regularized method, heavily benefiting from both priors and hierarchy.
Revised: Given this insight, we introduce A ttentive P riors for E xpressive and Transferable S kills (APES), a hierarchical KL-regularized method, heavily benefiting from both priors and hierarchy.

Original: Unlike existing approaches,
Revised: Unlike existing approaches,

Original: APES automates the choice of asymmetry by learning it in a data-driven, domaindependent, way based on our expressivity-transferability theorems.
Revised: APES automates the choice of asymmetry by learning it in a data-driven, domaindependent, way based on our expressivity-transferability theorems.

Original: Experiments over complex transfer domains of varying levels of extrapolation and sparsity, such as robot block stacking, demonstrate the criticality of the correct asymmetric choice, with APES drastically outperforming previous methods.
Revised: Experiments over complex transfer domains of varying levels of extrapolation and sparsity, such as robot block stacking, demonstrate the criticality of the correct asymmetric choice, with APES drastically outperforming previous methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: While Reinforcement Learning (RL) algorithms recently achieved impressive feats across a range of domains (Silver et al., 2017; Mnih et al., 2015; Lillicrap et al., 2015), they remain sample inefficient (Abdolmaleki et al., 2018; Haarnoja et al., 2018b) and are therefore of limited use for real-world robotics applications.
Revised: While Reinforcement Learning (RL) algorithms recently achieved impressive feats across a range of domains (Silver et al., 2017; Mnih et al., 2015; Lillicrap et al., 2015) they remain sample inefficient (Abdolmaleki et al., 2018; Haarnoja et al., 2018b) and are therefore of limited use for real-world robotics applications.

Original: Intelligent agents during their lifetime discover and reuse skills at multiple levels of behavioural and temporal abstraction to efficiently tackle new situations.
Revised: Intelligent agents during their lifetime discover and reuse skills at multiple levels of behavioural and temporal abstraction to efficiently tackle new situations.

Original: For example, in manipulation domains, beneficial abstractions could include low-level instantaneous motor primitives as well as higher-level object manipulation strategies.
Revised: For example, in manipulation domains, beneficial abstractions could include low-level instantaneous motor primitives as well as higher-level object manipulation strategies.

Original: Endowing lifelong learning RL agents (Parisi et al., 2019) with a similar ability could be vital towards attaining comparable sample efficiency.
Revised: Endowing lifelong learning RL agents (Parisi et al., 2019) with a similar ability could be vital towards attaining comparable sample efficiency.

Original: To this end, two paradigms have recently been introduced.
Revised: To this end, two paradigms have recently been introduced.

Original: KL-regularized RL (Teh et al., 2017;
Revised: KL-regularized RL (Teh et al., 2017;

Original: Galashov et al., 2019) presents an intuitive approach for automating skill reuse in multi-task learning.
Revised: Galashov et al., 2019) presents an intuitive approach for automating skill reuse in multi-task learning.

Original: By regularizing policy behaviour against a learnt task-agnostic prior, common behaviours across tasks are distilled into the prior, which encourages their reuse.
Revised: By regularizing policy behaviour against a learnt task-agnostic prior, common behaviours across tasks are distilled into the prior, which encourages their reuse.

Original: Concurrently, hierarchical RL also enables skill discovery (Wulfmeier et al., 2019; Merel et al., 2020; Hausman et al., 2018; Haarnoja et al., 2018a; Wulfmeier et al., 2020) by considering a two-level hierarchy in which the high-level policy is task-conditioned, whilst the low-level remains task-agnostic.
Revised: Concurrently, hierarchical RL also enables skill discovery (Wulfmeier et al., 2019; Merel et al., 2020; Hausman et al., 2018; Haarnoja et al., 2018a; Wulfmeier et al., 2020) by considering a two-level hierarchy in which the high-level policy is task-conditioned, whilst the low-level remains task-agnostic.

Original: The lower level of the hierarchy therefore also discovers skills that are transferable across tasks.
Revised: The lower level of the hierarchy therefore also discovers skills that are transferable across tasks.

Original: Both hierarchy and priors offer their own skill abstraction.
Revised: Both hierarchy and priors offer their own skill abstraction.

Original: However, when combined, hierarchical KL-regularized RL can discover multiple abstractions.
Revised: However, when combined, hierarchical KL-regularized RL can discover multiple abstractions.

Original: Whilst prior methods attempted this (Tirumala et al., 2019; 2020; Liu et al., 2022; Goyal et al., 2019), the transfer benefits from learning both abstractions varied drastically, with approaches like Tirumala et al.
Revised: Whilst prior methods attempted this (Tirumala et al., 2019; 2020; Liu et al., 2022; Goyal et al., 2019), the transfer benefits from learning both abstractions varied drastically, with approaches like Tirumala et al.

Original: (2019) unable to yield performance gains.
Revised: (2019) unable to yield performance gains.

Original: In fact, successful transfer of the aforementioned hierarchical and KL-regularized methods critically depends on the correct choice of information asymmetry (IA).
Revised: In fact, successful transfer of the aforementioned hierarchical and KL-regularized methods critically depends on the correct choice of information asymmetry (IA).

Original: IA more generally refers to an asymmetric masking of information across architectural modules.
Revised: IA more generally refers to an asymmetric masking of information across architectural modules.

Original: This masking forces independence to, and ideally generalisation across, the masked dimensions (Galashov et al., 2019).
Revised: This masking forces independence to, and ideally generalisation across, the masked dimensions (Galashov et al., 2019).

Original: For example, for self-driving cars, by conditioning the prior only on proprioceptive information it discovers skills independent to, and shared across, global coordinate frames.
Revised: For example, for self-driving cars, by conditioning the prior only on proprioceptive information it discovers skills independent to, and shared across, global coordinate frames.

Original: For manipulation, by not conditioning on certain object information, such as shape or weight, the robot learns generalisable grasping independent to these factors.
Revised: For manipulation, by not conditioning on certain object information, such as shape or weight, the robot learns generalisable grasping independent to these factors.

Original: Therefore, IA biases learnt behaviours and how they transfer across environments.
Revised: Therefore, IA biases learnt behaviours and how they transfer across environments.

Original: Previous works predefined their IAs, which were primarily chosen on intuition and independent of domain.
Revised: Previous works predefined their IAs, which were primarily chosen on intuition and independent of domain.

Original: In addition, previously explored asymmetries were narrow (Table 1), which if sub-optimal, limit transfer benefits.
Revised: In addition, previously explored asymmetries were narrow (Table 1), which if sub-optimal, limit transfer benefits.

Original: We demonstrate that this indeed is the case for many methods on our domains (Galashov et al., 2019; Bagatella et al., 2022; Tirumala et al., 2019; 2020; Pertsch et al., 2021; Wulfmeier et al., 2019).
Revised: We demonstrate that this indeed is the case for many methods on our domains (Galashov et al., 2019; Bagatella et al., 2022; Tirumala et al., 2019; 2020; Pertsch et al., 2021; Wulfmeier et al., 2019).

Original: A more systematic, theoretically and data driven, domain dependent, approach for choosing IA is thus required to maximally benefit from skills for transfer learning.
Revised: A more systematic, theoretically and data driven, domain dependent, approach for choosing IA is thus required to maximally benefit from skills for transfer learning.

Original: In this paper, we employ hierarchical KL-regularized RL to effectively transfer skills across sequential tasks.
Revised: In this paper, we employ hierarchical KL-regularized RL to effectively transfer skills across sequential tasks.

Original: We begin by theoretically and empirically showing the crucial expressivity-transferability trade-off, controlled by choice of IA, of skills across sequential tasks for hierarchical KL-regularized
Revised: We begin by theoretically and empirically showing the crucial expressivity-transferability trade-off, controlled by choice of IA, of skills across sequential tasks for hierarchical KL-regularized

Original: RL.
Revised: RL.

Original: Our expressivity-transferability theorems state that conditioning skill modules on too little or too much information, such as the current observation or entire history, can both be detrimental for transfer, due to the discovery of skills that are either too general (e.g. motor primitives) or too specialised (e.g. non-transferable task-level).
Revised: Our expressivity-transferability theorems state that conditioning skill modules on too little or too much information, such as the current observation or entire history, can both be detrimental for transfer, due to the discovery of skills that are either too general (e.g. motor primitives) or too specialised (e.g. non-transferable task-level).

Original: We demonstrate this by ablating over a wide range of asymmetries between the hierarchical policy and prior.
Revised: We demonstrate this by ablating over a wide range of asymmetries between the hierarchical policy and prior.

Original: We show the inefficiencies of previous methods that choose highly sub-optimal IAs for our domains, drastically limiting transfer performance.
Revised: We show the inefficiencies of previous methods that choose highly sub-optimal IAs for our domains, drastically limiting transfer performance.

Original: Given this insight, we introduce APES , ‘ A ttentive P riors for E xpressive and Transferable S kills’ as a method that forgoes user intuition and automates the choice of IA in a data driven, domain dependent, manner.
Revised: Given this insight, we introduce APES , ‘ A ttentive P riors for E xpressive and Transferable S kills’ as a method that forgoes user intuition and automates the choice of IA in a data driven, domain dependent, manner.

Original: APES builds on our expressivity-transferability theorems to learn the choice of asymmetry between policy and prior.
Revised: APES builds on our expressivity-transferability theorems to learn the choice of asymmetry between policy and prior.

Original: Specifically, APES conditions the prior on the entire history, allowing for expressive skills to be discovered, and learns a low-entropic attention-map over the input, paying attention only where necessary, to minimise covariate shift and improve transferability across domains.
Revised: Specifically, APES conditions the prior on the entire history, allowing for expressive skills to be discovered, and learns a low-entropic attention-mask over the input, paying attention only where necessary, to minimise covariate shift and improve transferability across domains.

Original: Experiments over domains of varying levels of sparsity and extrapolation, including a complex robot block stacking one, demonstrate APES’ consistent superior performance over existing methods, whilst automating IA choice and by-passing arduous IA sweeps.
Revised: Experiments over domains of varying levels of sparsity and extrapolation, including a complex robot block stacking one, demonstrate APES’ consistent superior performance over existing methods, whilst automating IA choice and by-passing arduous IA sweeps.

Original: Further ablations show the importance of combining hierarchy and priors for discovering expressive multi-modal behaviours.
Revised: Further ablations show the importance of combining hierarchy and priors for discovering expressive multi-modal behaviours.

Original:  S KILL T RANSFER IN R EINFORCEMENT L EARNING
Revised:  S KILL T RANSFER IN R EINFORCEMENT L EARNING

Original: We consider multi-task reinforcement learning in Partially Observable Markov Decision Processes (POMDPs), defined by M k = ( S , X , A , r k , p, p 0 k , γ ) , with tasks k sampled from p ( K ) .
Revised: We consider multi-task reinforcement learning in Partially Observable Markov Decision Processes (POMDPs), defined by M k = ( S , X , A , r k , p, p 0 k , γ ) , with tasks k sampled from p ( K ) .

Original: S , A , X denote observation, action, and history spaces.
Revised: S , A , X denote observation, action, and history spaces.

Original: p ( x ′ | x , a ) :
Revised: p ( x ′ | x , a ) :

Original: X × X ×
Revised: X × X ×

Original: A → R ≥ 0 is the dynamics model.
Revised: A → R ≥ 0 is the dynamics model.

Original: We denote the history of observations s ∈ S , actions a ∈
Revised: We denote the history of observations s ∈ S , actions a ∈

Original: A up to timestep t as x t =
Revised: A up to timestep t as x t =

Original: ( s 0 , a 0 , s 1 , a 1 , . . .
Revised: ( s 0 , a 0 , s 1 , a 1 , . . .

Original: , s t ) .
Revised: , s t ) .

Original: Reward function r k : X ×A×K → R is history-, action- and task-dependent.
Revised: Reward function r k : X ×A×K → R is history-, action- and task-dependent.

Original: 2.  KL-R EGULARIZED R EINFORCEMENT L EARNING
Revised: 2. H IERARCHICAL KL-R EGULARIZED R EINFORCEMENT L EARNING

Original: The typical multi-task KL-regularized RL objective (Todorov, 2007; Kappen et al., 2012; Rawlik et al., 2012; Schulman et al., 2017) takes the form:
Revised: The typical multi-task KL-regularized RL objective (Todorov, 2007; Kappen et al., 2012; Rawlik et al., 2012; Schulman et al., 2017) takes the form:

Original: [Equation]
Revised: [Equation]

Original: (cid:19)(cid:35) (1) where γ is the discount factor and α 0 weighs the individual objective terms.
Revised: (1) where γ is the discount factor and α 0 weighs the individual objective terms.

Original: π and π 0 denote the task-conditioned policy and task-agnostic prior respectively.
Revised: π and π 0 denote the task-conditioned policy and task-agnostic prior respectively.

Original: The expectation is taken over tasks and trajectories τ from policy π and initial observation distribution p 

Revised: The expectation is taken over tasks and trajectories τ from policy π and initial observation distribution p

Original: k ( s 0 ) , (i.e. p π ( τ ) ).
Revised: k ( s 0 ) , (i.e. p π ( τ ) ).

Original: Summation over t occurs across all episodic timesteps.
Revised: Summation over t occurs across all episodic timesteps.

Original: When optimised with respect to π , this objective can be viewed as a trade-off between maximising rewards whilst remaining close to trajectories produced by π 0 .
Revised: When optimised with respect to π , this objective can be viewed as a trade-off between maximising rewards whilst remaining close to trajectories produced by π 0 .

Original: When π 0 is learnt, it can learn shared behaviours across tasks and bias multi-task exploration (Teh et al., 2017).
Revised: When π 0 is learnt, it can learn shared behaviours across tasks and bias multi-task exploration (Teh et al., 2017).

Original: We consider the sequential learning paradigm, where skills are learnt from past tasks, p source ( K ) , and leveraged while attempting the transfer set of tasks, p trans ( K ) .
Revised: We consider the sequential learning paradigm, where skills are learnt from past tasks, p source ( K ) , and leveraged while attempting the transfer set of tasks, p trans ( K ) .

Original: 2. H IERARCHICAL KL-R EGULARIZED R EINFORCEMENT L EARNING

Original: While KL-regularized RL has achieved success across various settings (Abdolmaleki et al., 2018;
Revised: While KL-regularized RL has achieved success across various settings (Abdolmaleki et al., 2018;

Original: Teh et al., 2017; Pertsch et al., 2020;
Revised: Teh et al., 2017; Pertsch et al., 2020;

Original: Haarnoja et al., 2018a), recently Tirumala et al.
Revised: Haarnoja et al., 2018a), recently Tirumala et al.

Original: (2019) proposed a hierarchical extension where policy π and prior π 0 are augmented with latent variables, π ( a , z | x , k ) =
Revised: (2019) proposed a hierarchical extension where policy π and prior π 0 are augmented with latent variables, π ( a , z | x , k ) =

Original: π H ( z | x , k ) π L ( a | z , x ) and π 0 ( a , z | x ) =
Revised: π H ( z | x , k ) π L ( a | z , x ) and π 0 ( a , z | x ) =

Original: π H 

Revised: π H

Original: ( z | x ) π L 0 ( a | z , x ) , where subscripts H and L denote the higher and lower hierarchical levels.
Revised: ( z | x ) π L 0 ( a | z , x ) , where subscripts H and L denote the higher and lower hierarchical levels.

Original: This structure encourages the shared low-level policy ( π L = π L 0 ) to discover task-agnostic behavioural primitives, whilst the high-level discovers higher-level task relevant skills.
Revised: This structure encourages the shared low-level policy ( π L = π L 0 ) to discover task-agnostic behavioural primitives, whilst the high-level discovers higher-level task relevant skills.

Original: By not conditioning the high-level prior on task-id, Tirumala et al.
Revised: By not conditioning the high-level prior on task-id, Tirumala et al.

Original: (2019) encourage the reuse of common high-level abstractions across tasks.
Revised: (2019) encourage the reuse of common high-level abstractions across tasks.

Original: They also propose the following upper bound for approximating the KL-divergence between hierarchical policy and prior:
Revised: They also propose the following upper bound for approximating the KL-divergence between hierarchical policy and prior:

Original: [Equation]
Revised: [Equation]

Original: We omit task conditioning and declaring shared modules to show this bound is agnostic to this.
Revised: We omit task conditioning and declaring shared modules to show this bound is agnostic to this.

Original: 2. I NFORMATION A SYMMETRY
Revised: 2. I NFORMATION A SYMMETRY

Original: [Table]
Revised: [Table]

Original: Information Asymmetry (IA) is a key component in both of the aforementioned approaches, promoting the discovery of behaviours that generalise.
Revised: Information Asymmetry (IA) is a key component in both of the aforementioned approaches, promoting the discovery of behaviours that generalise.

Original: IA can be understood as the masking of information accessible by certain modules.
Revised: IA can be understood as the masking of information accessible by certain modules.

Original: Not conditioning on specific environment aspects forces
Revised: Not conditioning on specific environment aspects forces

Original: independence and generalisation across them (Galashov et al., 2019).
Revised: independence and generalisation across them (Galashov et al., 2019).

Original: In the context of (hierarchical)
Revised: In the context of (hierarchical)

Original: KL-regularized RL, the explored asymmetries between the (high-level) policy, π ( H ) , and prior, π ( H ), have been narrow (Tirumala et al., 2019; 2020; Liu et al., 2022; Pertsch et al., 2020; 2021;
Revised: KL-regularized RL, the explored asymmetries between the (high-level) policy, π ( H ) , and prior, π ( H ), have been narrow (Tirumala et al., 2019; 2020; Liu et al., 2022; Pertsch et al., 2020; 2021;

Original: Rao et al., 2021; Ajay et al., 2020; Goyal et al., 2019).
Revised: Rao et al., 2021; Ajay et al., 2020; Goyal et al., 2019).

Original: Concurrent with our research, Bagatella et al.
Revised: Concurrent with our research, Bagatella et al.

Original: (2022) published work exploring a wider range of asymmetries, closer to those we explore.
Revised: (2022) published work exploring a wider range of asymmetries, closer to those we explore.

Original: We summarise explored asymmetries in Table 1 (with a t − n : t representing action history up to n steps in the past).
Revised: We summarise explored asymmetries in Table 1 (with a t − n : t representing action history up to n steps in the past).

Original: Choice of information conditioning heavily influences which skills can be uncovered and how well they transfer.
Revised: Choice of information conditioning heavily influences which skills can be uncovered and how well they transfer.

Original: For example, Pertsch et al.
Revised: For example, Pertsch et al.

Original: (2020) discover observation-dependent behaviours, such as navigating corridors in maze environments, yet are unable to learn history-dependent skills, such as never traversing the same corridor twice.
Revised: (2020) discover observation-dependent behaviours, such as navigating corridors in maze environments, yet are unable to learn history-dependent skills, such as never traversing the same corridor twice.

Original: In contrast, Liu et al.
Revised: In contrast, Liu et al.

Original: (2022), by conditioning on history, are able to learn these behaviours.
Revised: (2022), by conditioning on history, are able to learn these behaviours.

Original: However, as we will show, in many scenarios, na ¨ ıvely conditioning on entire history can be detrimental for transfer, by discovering behaviours that do not generalise favourably across history instances, between tasks.
Revised: However, as we will show, in many scenarios, na ¨ ıvely conditioning on entire history can be detrimental for transfer, by discovering behaviours that do not generalise favourably across history instances, between tasks.

Original: We refer to this dilemma as the expressivitytransferability trade-off.
Revised: We refer to this dilemma as the expressivitytransferability trade-off.

Original: Crucially, all previous works predefine the choice of asymmetry, based on the practitioner’s intuition, that may be sub-optimal for skill transfer.
Revised: Crucially, all previous works predefine the choice of asymmetry, based on the practitioner’s intuition, that may be sub-optimal for skill transfer.

Original: By introducing theory behind the expressivity-transferability of skills, we present a simple data-driven method for automating the choice of IA, by learning it, yielding transfer benefits.
Revised: By introducing theory behind the expressivity-transferability of skills, we present a simple data-driven method for automating the choice of IA, by learning it, yielding transfer benefits.

Original:  M ODEL A RCHITECTURE AND THE E XPRESSIVITY -T RANSFERABILITY T RADE -O FF
Revised:  M ODEL A RCHITECTURE AND THE E XPRESSIVITY -T RANSFERABILITY T RADE -O FF

Original: [Figure]
Revised: [Figure]

Original: The hierarchical policy modules π H and π L are regularized against their corresponding prior modules π Hi and π Li  inputs to  filtered  gating function  colored rectangle.
Revised: The hierarchical policy modules π H and π L are regularized against their corresponding prior modules π Hi and π Li . The inputs to each module are filtered by an information gating function (IGF), depicted with colored rectangles.

Original: To rigorously investigate the contribution of priors, hierarchy, and information asymmetry on skill transfer, it is important to isolate each individual mechanism while enabling the recovery of previous models of interest.
Revised: To rigorously investigate the contribution of priors, hierarchy, and information asymmetry for skill transfer, it is important to isolate each individual mechanism while enabling the recovery of previous models of interest.

Original: To this end, we present the unified architecture in Fig.
Revised: To this end, we present the unified architecture in Fig.

Original: 1, which introduces information gating functions (IGFs) as a means of decoupling IA from architecture.
Revised: 1, which introduces information gating functions (IGFs) as a means of decoupling IA from architecture.

Original: Each component has its own IGF, depicted with  colored rectangle.
Revised: Each component has its own IGF, depicted with a colored rectangle.

Original: Every module is fed all environment information x k = ( x , k ) and distinctly chosen IGFs mask which part of the input each network has access to, thereby influencing which skills they learn.
Revised: Every module is fed all environment information x k = ( x , k ) and distinctly chosen IGFs mask which part of the input each network has access to, thereby influencing which skills they learn.

Original: By presenting multiple priors, we enable a comparison with existing literature.
Revised: By presenting multiple priors, we enable a comparison with existing literature.

Original: With the right masking, one can recover previously investigated asymmetries (Tirumala et al., 2019; 2020;
Revised: With the right masking, one can recover previously investigated asymmetries (Tirumala et al., 2019; 2020;

Original: Pertsch et al., 2020; Bagatella et al., 2022;
Revised: Pertsch et al., 2020; Bagatella et al., 2022;

Original: Goyal et al., 2019), explore additional ones, and also express purely hierarchical (Wulfmeier et al., 2019) and KL-regularized equivalents (Galashov et al., 2019; Haarnoja et al., 2018c).
Revised: Goyal et al., 2019), explore additional ones, and also express purely hierarchical (Wulfmeier et al., 2019) and KL-regularized equivalents (Galashov et al., 2019; Haarnoja et al., 2018c).

Original: 3. T HE I NFORMATION A SYMMETRY E XPRESSIVITY - TRANSFERABILITY T RADE -O FF
Revised: 3. T HE I NFORMATION A SYMMETRY E XPRESSIVITY - TRANSFERABILITY T RADE -O FF

Original: While existing works investigating the role of IAs for skill transfer in hierarchical KL-regularized RL have focused on multi-task learning (Galashov et al., 2019) 1 , we focus on the sequential task setting, in particular the prior’s π 0 ability to handle covariate shift.
Revised: While existing works investigating the role of IAs for skill transfer in hierarchical KL-regularized RL have focused on multi-task learning (Galashov et al., 2019) 1 , we focus on the sequential task setting, in particular the prior’s π 0 ability to handle covariate shift.

Original: In contrast to multi-task learning, in the sequential setting, there exists abrupt distributional shifts, during training, for task p
Revised: In contrast to multi-task learning, in the sequential setting, there exists abrupt distributional shifts, during training, for task p

Original: ( K ) and trajectory p π ( τ ) distributions.
Revised: ( K ) and trajectory p π ( τ ) distributions.

Original: As such, it is important that the prior handles such distributional and 
Revised: As such, it is important that the prior handles such distributional and covariate

Original: [Figure]
Revised: [Figure]

Original: H a ll w a y
Revised: H a ll w a y

Original: Corridor Depth
Revised: Corridor Depth

Original: C o
Revised: C o

Original: rr i
Revised: rr i

Original: d o r N u
Revised: d o r N u

Original: m b e r ( 5 )covariate shifts (see Theorem 3.1 for  definition).
Revised: m b e r ( 5 ) shifts (see Theorem 3.1 for a definition).

Original: For multi-task learning, trajectory shifts are gradual and skills are continually retrained over such shifts, alleviating transfer issues.
Revised: For multi-task learning, trajectory shifts are gradual and skills are continually retrained over such shifts, alleviating transfer issues.

Original: In general, IA plays a crucial role, influencing the level of covariate shift encountered by the prior during learning:
Revised: In general, IA plays a crucial role, influencing the level of covariate shift encountered by the prior during learning:

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: The more random variables a network depends on, the larger the covariate shift (input distributional shift, here represented by KL-divergence) encountered across sequential tasks.
Revised: The more random variables a network depends on, the larger the covariate shift (input distributional shift, here represented by KL-divergence) encountered across sequential tasks.

Original: That is, for distributions p , q and inputs b , c such that b = ( b 0 , b 1 , ..., b n ) and c ⊂ b :
Revised: That is, for distributions p , q and inputs b , c such that b = ( b 0 , b 1 , ..., b n ) and c ⊂ b :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Appendix B.1.
Revised: See Appendix B.1.

Original: In our case, p and q can be interpreted as training p source ( · ) and transfer p trans ( · ) distributions over network inputs, such as history x t for high-level prior π H 0 .
Revised: In our case, p and q can be interpreted as training p source ( · ) and transfer p trans ( · ) distributions over network inputs, such as history x t for high-level prior π H 0 .

Original: Intuitively, Theorem 3.1 states that the more variables you condition your network on, the less likely it will transfer due to increased covariate shift encountered between source and transfer domains, thus promoting minimal information conditioning.
Revised: Intuitively, Theorem 3.1 states that the more variables you condition your network on, the less likely it will transfer due to increased covariate shift encountered between source and transfer domains, thus promoting minimal information conditioning.

Original: For example, imagine conditioning the high-level prior on either the entire history x 0: t or a subset of it x t − n : t , n ∈ [0 , t − 1] (the subscript referring to the range of history values).
Revised: For example, imagine conditioning the high-level prior on either the entire history x 0: t or a subset of it x t − n : t , n ∈ [0 , t − 1] (the subscript referring to the range of history values).

Original: According to Theorem 3.1, the covariate shift across sequential tasks will be smaller if we condition on a subset of the history, D KL ( p source ( x 0: t ) ∥
Revised: According to Theorem 3.1, the covariate shift across sequential tasks will be smaller if we condition on a subset of the history, D KL ( p source ( x 0: t ) ∥

Original: p trans ( x 0: t )) ≥ D KL ( p source ( x t − n : t ) ∥
Revised: p trans ( x 0: t )) ≥ D KL ( p source ( x t − n : t ) ∥

Original: p trans ( x t − n : t )) .
Revised: p trans ( x t − n : t )) .

Original: Interestingly, covariate shift is upper-bounded by trajectory shift: D KL ( p π source ( τ ) ∥
Revised: Interestingly, covariate shift is upper-bounded by trajectory shift: D KL ( p π source ( τ ) ∥

Original: p π trans ( τ )) ≥
Revised: p π trans ( τ )) ≥

Original: D KL ( p π source ( τ f ) ∥
Revised: D KL ( p π source ( τ f ) ∥

Original: p π trans ( τ f ))
Revised: p π trans ( τ f ))

Original: (using Theorem 3.1), with the right hand side representing covariate shift over network inputs τ f = IGF ( τ ) , filtered trajectories (e.g. τ f = x t − n : t , τ f ⊂ τ ), and π source , π trans , source and transfer domain policies.
Revised: (using Theorem 3.1), with the right hand side representing covariate shift over network inputs τ f = IGF ( τ ) , filtered trajectories (e.g. τ f = x t − n : t , τ f ⊂ τ ), and π source , π trans , source and transfer domain policies.

Original: It is therefore crucial, if possible, to minimise both trajectory and covariate shifts across domains, to benefit from previous skills.
Revised: It is therefore crucial, if possible, to minimise both trajectory and covariate shifts across domains, to benefit from previous skills.

Original: Nevertheless, the less information a prior is conditioned on, the less knowledge that can be distilled and transferred:
Revised: Nevertheless, the less information a prior is conditioned on, the less knowledge that can be distilled and transferred:

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: The more random variables a network depends on, the greater its ability to distil knowledge in the expectation (output distributional shift between network and target distribution, here represented by the expected KL-divergence).
Revised: The more random variables a network depends on, the greater its ability to distil knowledge in the expectation (output distributional shift between network and target distribution, here represented by the expected KL-divergence).

Original: That is, for target distribution p and network q with outputs a and possible inputs b , c , d , such that b = ( b 0 , b 1 , ..., b n ) , d ⊂ c
Revised: That is, for target distribution p and network q with outputs a and possible inputs b , c , d , such that b = ( b 0 , b 1 , ..., b n ) , d [Equation] c

Original: [Equation]
Revised: [Equation]

Original: b ,
Revised: b ,

Original: [Equation]
Revised: [Equation]

Original: d
Revised: c :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Appendix B.2.
Revised: See Appendix B.2.

Original: In this particular instance, p and q could be interpreted as policy π and prior π 0 distributions, a as action a t , b as history x 0: t , and c , d , e as subsets of the history (e.g. x t − n : t , x t − m : t , x t − n : t − m respectively, with n > m and m & n ∈ [0 , t ] ), with e denoting the set of variables in c but not d .
Revised: In this particular instance, p and q could be interpreted as policy π and prior π 0 distributions, a as action a t , b as history x 0: t , and c , d , e as subsets of the history (e.g. x t − n : t , x t − m : t , x t − n : t − m respectively, with n > m and m & n ∈ [0 , t ] ), with e denoting the set of variables in c but not d .

Original: Intuitively,
Revised: Intuitively,

Original: Theorem 3.2 states in the expectation, conditioning on more information improves knowledge distillation between policy and prior (e.g. E π 0 ( x t − n : t − m | x t − m : t )
Revised: Theorem 3.2 states in the expectation, conditioning on more information improves knowledge distillation between policy and prior (e.g. E π 0 ( x t − n : t − m | x t − m : t )

Original: [D KL ( π ( a t | x 0: t ) ∥
Revised: [D KL ( π ( a t | x 0: t ) ∥

Original: π 0 ( a t | x t − n : t ))] ≤
Revised: π 0 ( a t | x t − n : t ))] ≤

Original: D KL ( π ( a t | x 0: t ) ∥
Revised: D KL ( π ( a t | x 0: t ) ∥

Original: π 0 ( a t | x t − m : t )) , with π 0 ( x t − n : t − m | x t − m : t ) the conditional distribution, induced by π 0 , of history subset x t − n : t − m given x t − m : t ).
Revised: π 0 ( a t | x t − m : t )) , with π 0 ( x t − n : t − m | x t − m : t ) the conditional distribution, induced by π 0 , of history subset x t − n : t − m given x t − m : t ).

Original: Therefore, IA leads to an expressivity-transferability trade-off of skills (Theorems 3.1 and 3.2).
Revised: Therefore, IA leads to an expressivity-transferability trade-off of skills (Theorems 3.1 and 3.2).

Original: Interestingly, hierarchy does not influence covariate shift and hence does not hurt transferability, but it does increase network expressivity (e.g. of the prior), enabling the distillation and transfer of rich multi-modal behaviours present in the real-world.
Revised: Interestingly, hierarchy does not influence covariate shift and hence does not hurt transferability, but it does increase network expressivity (e.g. of the prior), enabling the distillation and transfer of rich multi-modal behaviours present in the real-world.

Original:  APES: A TTENTIVE P RIORS FOR E XPRESSIVE AND T RANSFERABLE S KILLS
Revised:  APES: A TTENTIVE P RIORS FOR E XPRESSIVE AND T RANSFERABLE S KILLS

Original: While previous works chose IA on intuition (Tirumala et al., 2019; 2020; Galashov et al., 2019;
Revised: While previous works chose IA on intuition (Tirumala et al., 2019; 2020; Galashov et al., 2019;

Original: Pertsch et al., 2020; Wulfmeier et al., 2019; Bagatella et al., 2022; Singh et al., 2020; Ajay et al., 2020; Liu et al., 2022) we propose learning it.
Revised: Pertsch et al., 2020; Wulfmeier et al., 2019; Bagatella et al., 2022; Singh et al., 2020; Ajay et al., 2020; Liu et al., 2022) we propose learning it.

Original: Consider the information gating functions (IGFs) introduced in Section 3 and depicted in Figure 1.
Revised: Consider the information gating functions (IGFs) introduced in Section 3 and depicted in Figure 1.

Original: Existing methods can be recovered by having the
Revised: Existing methods can be recovered by having the

Original: IGFs perform hard attention: IGF ( x k )
Revised: IGFs perform hard attention: IGF ( x k )

Original: = m
Revised: = m

Original: ⊙ x k , with m ∈ { 0 , 1 } dim ( x k ) , predefined and static, and ⊙ representing element-wize multiplication.
Revised: ⊙ x k , with m ∈ { 0 , 1 } dim ( x k ) , predefined and static, and ⊙ representing element-wize multiplication.

Original: In contrast, we propose performing soft attention with m ∈ [0 , 1] dim ( x k ) and learn m based on: 1) the hierarchical KL-regularized RL objective
Revised: In contrast, we propose performing soft attention with m ∈ [0 , 1] dim ( x k ) and learn m based on: 1) the hierarchical KL-regularized RL objective

Original: (Equations (1) and (2)); 2) L IGF ( m ) =
Revised: (Equations (1) and (2)); 2) L IGF ( m ) =

Original: −H ( m ) , H denoting entropy (calculated by turning m into a probability distribution by performing Softmax over it), thereby encouraging low entropic, sparse
Revised: −H ( m ) , H denoting entropy (calculated by turning m into a probability distribution by performing Softmax over it), thereby encouraging low entropic, sparse

Original: IGFs (similar to Salter et al.
Revised: IGFs (similar to Salter et al.

Original: (2019) applying a related technique for sim2real transfer):
Revised: (2019) applying a related technique for sim2real transfer):

Original: [Equation]
Revised: [Equation]

Original: With α m i weighing the relative importance of the entropy and KL-regularized RL objectives for each attention mask { m i } i ∈ I for each module using self-attention (e.g. π H 0 ), I denoting this set.
Revised: With α m i weighing the relative importance of the entropy and KL-regularized RL objectives for each attention mask { m i } i ∈ I for each module using self-attention (e.g. π H 0 ), I denoting this set.

Original: Whilst soft attention does not eliminate dimensions in the same way that hard attention does, thus losing the strict connection with Theorems 3.1 and 3.2, in practice it often leads to many 0 -attention elements (Salter et al., 2020; Mott et al., 2019).
Revised: Whilst soft attention does not eliminate dimensions in the same way that hard attention does, thus losing the strict connection with Theorems 3.1 and 3.2, in practice it often leads to many 0 -attention elements (Salter et al., 2020; Mott et al., 2019).

Original: m i spans all history dimensions and supports intra-observation and intra-action attention.
Revised: m i spans all history dimensions and supports intra-observation and intra-action attention.

Original: We train off-policy akin to SAC (Haarnoja et al., 2018b), sampling experiences from the replay buffer, approximating the return of the agent using Retrace (Munos et al., 2016) and double Q-learning (Hasselt, 2010) to train our critic.
Revised: We train off-policy akin to SAC (Haarnoja et al., 2018b), sampling experiences from the replay buffer, approximating the return of the agent using Retrace (Munos et al., 2016) and double Q-learning (Hasselt, 2010) to train our critic.

Original: Refer to Appendices A and D for full training details.
Revised: Refer to Appendices A and D for full training details.

Original: Exposing IGFs to all available information x k , we enable expressive skills that maximize the KL-regularized RL objective, with complex, potentially long-range, temporal dependencies (Theorem 3.2).
Revised: Exposing IGFs to all available information x k , we enable expressive skills that maximize the KL-regularized RL objective, with complex, potentially long-range, temporal dependencies (Theorem 3.2).

Original: Encouraging low-entropic masks m i promotes minimal information conditioning (by limiting the IGF’s channel capacity) whilst still capturing expressive behaviours.
Revised: Encouraging low-entropic masks m i promotes minimal information conditioning (by limiting the IGF’s channel capacity) whilst still capturing expressive behaviours.

Original: This is achieved by paying attention only where necessary to key environment aspects (Salter et al., 2022) that are crucial for decision making and hence heavily influence behaviour expressivity.
Revised: This is achieved by paying attention only where necessary to key environment aspects (Salter et al., 2022) that are crucial for decision making and hence heavily influence behaviour expressivity.

Original: Minimising the dependence on redundant information (aspects of the observation s , action a , or history x spaces that behaviours are independent to), we minimise covariate shift and improve the transferability of skills to downstream domains (Theorem 3.1).
Revised: Minimising the dependence on redundant information (aspects of the observation s , action a , or history x spaces that behaviours are independent to), we minimise covariate shift and improve the transferability of skills to downstream domains (Theorem 3.1).

Original: Consider learning the IGF of high-level prior π H 0 for a humanoid navigation task.
Revised: Consider learning the IGF of high-level prior π H 0 for a humanoid navigation task.

Original: Low-level skills π L could correspond to motor-primitives, whilst the high-level prior could represent navigation skills.
Revised: Low-level skills π L could correspond to motor-primitives, whilst the high-level prior could represent navigation skills.

Original: For navigation, joint quaternions are not relevant, but the Cartesian position is.
Revised: For navigation, joint quaternions are not relevant, but the Cartesian position is.

Original: By learning to mask parts of the observations corresponding to joints, the agent becomes invariant and robust to covariate shifts across these dimensions (unseen joint configurations).
Revised: By learning to mask parts of the observations corresponding to joints, the agent becomes invariant and robust to covariate shifts across these dimensions (unseen joint configurations).

Original: We call our method APES, ‘ A ttentive P riors for E xpressive and Transferable S kills’.
Revised: We call our method APES, ‘ A ttentive P riors for E xpressive and Transferable S kills’.

Original: 4. T RAINING R EGIME AND T HE I NFORMATION A SYMMETRY SETUP
Revised: 4. T RAINING R EGIME AND T HE I NFORMATION A SYMMETRY SETUP

Original: We are concerned with investigating the roles of priors, hierarchy and IA for transfer in sequential task learning, where skills learnt over past tasks p source ( K ) are leveraged for transfer tasks p trans ( K ) .
Revised: We are concerned with investigating the roles of priors, hierarchy and IA for transfer in sequential task learning, where skills learnt over past tasks p source ( K ) are leveraged for transfer tasks p trans ( K ) .

Original: While one can investigate IA between hierarchical levels ( π H , π L ) as well as between policy and prior ( π , π 0 ), we concern ourselves solely with the latter.
Revised: While one could investigate IA between hierarchical levels ( π H , π L ) as well as between policy and prior ( π , π 0 ), we concern ourselves solely with the latter.

Original: Specifically, to keep our comparisons with existing literature fair, we condition π L on s t and z t , and share it with the prior, π L = π L 0 , thus enabling expressive multi-modal behaviours to be discovered with respect to s t (Tirumala et al., 2019; 2020; Wulfmeier et al., 2019).
Revised: Specifically, to keep our comparisons with existing literature fair, we condition π L on s t and z t , and share it with the prior, π L = π L 0 , thus enabling expressive multi-modal behaviours to be discovered with respect to s t (Tirumala et al., 2019; 2020; Wulfmeier et al., 2019).

Original: In this paper, we focus on the role of IA between high-level policy π H and prior π H 0 for supporting expressive and transferable high-level skills between tasks.
Revised: In this paper, we focus on the role of IA between high-level policy π H and prior π H 0 for supporting expressive and transferable high-level skills between tasks.

Original: As is common, we assume the source tasks are solved before tackling the transfer tasks.
Revised: As is common, we assume the source tasks are solved before tackling the transfer tasks.

Original: Therefore, for analysis purposes it does not matter whether we learn skills from source domain demonstrations provided by a hardcoded expert or an optimal RL agent.
Revised: Therefore, for analysis purposes it does not matter whether we learn skills from source domain demonstrations provided by a hardcoded expert or an optimal RL agent.

Original: For simplicity, we discover skills and skill priors using variational behavioural cloning from expert policy π e samples:
Revised: For simplicity, we discover skills and skill priors using variational behavioural cloning from expert policy π e samples:

Original: [Equation]
Revised: [Equation]

Original: Equation (4) can be viewed as hierarchical KL-regularized RL in the absence of rewards and with two priors: the one we learn π 0 ; the other the expert π e .
Revised: Equation (4) can be viewed as hierarchical KL-regularized RL in the absence of rewards and with two priors: the one we learn π 0 ; the other the expert π e .

Original: See Appendix A.2 for a deeper discussion on the similarities with KL-regularized RL.
Revised: See Appendix A.2 for a deeper discussion on the similarities with KL-regularized RL.

Original: We then transfer the skills and solve the transfer domains using hierarchical KL-regularized RL (as per Equation (3)).
Revised: We then transfer the skills and solve the transfer domains using hierarchical KL-regularized RL (as per Equation (3)).

Original: To compare the influence of distinct IAs for transfer in a controlled way, we propose the following regime: Stage 1) train a single hierarchical policy π in the multi-task setup using Equation (4), but prevent gradient flow from prior π 0 to policy.
Revised: To compare the influence of distinct IAs for transfer in a controlled way, we propose the following regime: Stage 1) train a single hierarchical policy π in the multi-task setup using Equation (4), but prevent gradient flow from prior π 0 to policy.

Original: Simultaneously, based on the ablation, train distinct priors (with differing IAs) on Equation (4) to imitate the policy.
Revised: Simultaneously, based on the ablation, train distinct priors (with differing IAs) on Equation (4) to imitate the policy.

Original: As such, we compare various IAs influence on skill distillation and transfer in a controlled manner, as they all distil behaviours from the same policy; Stage 2) freeze the shared modules ( π L , π H 0 ) and train a newly instantiated π H on the transfer task.
Revised: As such, we compare various IAs influence on skill distillation and transfer in a controlled manner, as they all distil behaviours from the same policy; Stage 2) freeze the shared modules ( π L , π H 0 ) and train a newly instantiated π H on the transfer task.

Original: By freezing π L , we assume the low-level skills from the source domains suffice for the transfer domains, often called the modularity assumption (Khetarpal et al., 2020a; Salter et al., 2022).
Revised: By freezing π L , we assume the low-level skills from the source domains suffice for the transfer domains, often called the modularity assumption (Khetarpal et al., 2020a; Salter et al., 2022).

Original: While appearing restrictive,
Revised: While appearing restrictive,

Original: [Figure]
Revised: [Figure]

Original: the increasingly diverse the source domains are (commonly desired in settings like lifelong learning (Khetarpal et al., 2020a) and offline RL), the increasingly probable the optimal transfer policy can be obtained by recomposing the learnt skills.
Revised: the increasingly diverse the source domains are (commonly desired in settings like lifelong learning (Khetarpal et al., 2020a) and offline RL), the increasingly probable the optimal transfer policy can be obtained by recomposing the learnt skills.

Original: If this assumption does not hold, one could either fine-tune π L during transfer, which would require tackling the catastrophic forgetting of skills (Kirkpatrick et al., 2017), or train additional skills (by expanding z dimensionality for discrete z spaces).
Revised: If this assumption does not hold, one could either fine-tune π L during transfer, which would require tackling the catastrophic forgetting of skills (Kirkpatrick et al., 2017), or train additional skills (by expanding z dimensionality for discrete z spaces).

Original: We leave this as future work.
Revised: We leave this as future work.

Original: We also leave extending APES to sub-optimal demonstration learning as future work, potentially using advantage-weighted regression (Peng et al., 2019), rather than behavioral cloning, to learn skills.
Revised: We also leave extending APES to sub-optimal demonstration learning as future work, potentially using advantage-weighted regression (Peng et al., 2019), rather than behavioral cloning, to learn skills.

Original: For further details refer to Appendices A and D.
Revised: For further details refer to Appendices A and D.

Original:  E XPERIMENTS AND R ESULTS
Revised:  E XPERIMENTS AND R ESULTS

Original: Our experiments are designed to answer the following sequential task questions: (1) Can we benefit from both hierarchy and priors for effective transfer?
Revised: Our experiments are designed to answer the following sequential task questions: (1) Can we benefit from both hierarchy and priors for effective transfer?

Original: (2) How important is IA choice between high-level policy and prior.
Revised: (2) How important is IA choice between high-level policy and prior.

Original: Does it lead to an impactful expressivity-transferability trade-off?
Revised: Does it lead to an impactful expressivity-transferability trade-off?

Original: In practice, how detrimental is covariate shift for transfer?
Revised: In practice, how detrimental is covariate shift for transfer?

Original: (3) How favourably does APES automate the choice of IA for effective transfer?
Revised: (3) How favourably does APES automate the choice of IA for effective transfer?

Original: Which IAs are discovered?
Revised: Which IAs are discovered?

Original: (4) How important is hierarchy for transferring expressive skills?
Revised: (4) How important is hierarchy for transferring expressive skills?

Original: Is hierarchy necessary?
Revised: Is hierarchy necessary?

Original: We compare against competitive skill transfer baselines, primarily Tirumala et al.
Revised: We compare against competitive skill transfer baselines, primarily Tirumala et al.

Original: (2019; 2020); Pertsch et al.
Revised: (2019; 2020); Pertsch et al.

Original: Wulfmeier et al.
Revised: Wulfmeier et al.

Original: (2019), on similar navigation and manipulation tasks for which they were originally designed and tested against.
Revised: (2019), on similar navigation and manipulation tasks for which they were originally designed and tested against.

Original: 5. E NVIRONMENTS
Revised: 5. E NVIRONMENTS

Original: We evaluate on two domains: one designed for controlled investigation of core agent capabilities and the another, more practical, robotics domain (see Figure
Revised: We evaluate on two domains: one designed for controlled investigation of core agent capabilities and the another, more practical, robotics domain (see Figure

Original: Both exhibit modular behaviours whose discovery could yield transfer benefits.
Revised: Both exhibit modular behaviours whose discovery could yield transfer benefits.

Original: See Appendix C for full environmental setup details.
Revised: See Appendix C for full environmental setup details.

Original: • CorridorMaze.
Revised: • CorridorMaze.

Original: The agent must traverse corridors in a given ordering.
Revised: The agent must traverse corridors in a given ordering.

Original: We collect 4 ∗ 10 3 trajectories from a scripted policy traversing any random ordering of two corridors ( p source ( K ) ).
Revised: We collect 4 ∗ 10 3 trajectories from a scripted policy traversing any random ordering of two corridors ( p source ( K ) ).

Original: For transfer ( p trans ( K ) ), an inter- or extrapolated ordering must be traversed (number of sequential corridors = { 2 , 4 } ) allowing us to inspect the generalization ability of distinct priors to increasing levels of covariate shift.
Revised: For transfer ( p trans ( K ) ), an inter- or extrapolated ordering must be traversed (number of sequential corridors = { 2 , 4 } ) allowing us to inspect the generalization ability of distinct priors to increasing levels of covariate shift.

Original: We also investigate the influence of covariate shift on effective transfer across reward sparsity levels: s-sparse (short for semi-sparse), rewarding per half-corridor completion; sparse , rewarding on task completion.
Revised: We also investigate the influence of covariate shift on effective transfer across reward sparsity levels: s-sparse (short for semi-sparse), rewarding per half-corridor completion; sparse , rewarding on task completion.

Original: Our transfer tasks are sparse 2 corr and s-sparse 4 corr .
Revised: Our transfer tasks are sparse 2 corr and s-sparse 4 corr .

Original: • Stack.
Revised: • Stack.

Original: The agent must stack a subset of four blocks over a target pad in a given ordering.
Revised: The agent must stack a subset of four blocks over a target pad in a given ordering.

Original: The blocks have distinct masses and only lighter blocks should be placed on heavier ones.
Revised: The blocks have distinct masses and only lighter blocks should be placed on heavier ones.

Original: Therefore, discovering temporal behaviours corresponding to sequential block stacking according to mass, is beneficial.
Revised: Therefore, discovering temporal behaviours corresponding to sequential block stacking according to mass, is beneficial.

Original: We collect 17 .
Revised: We collect 17 .

Original: 5 ∗ 10 3 trajectories from a scripted policy, stacking any two blocks given this requirement ( p source ( K ) ).
Revised: 5 ∗ 10 3 trajectories from a scripted policy, stacking any two blocks given this requirement ( p source ( K ) ).

Original: The extrapolated transfer task ( p trans ( K ) ), called 4 blocks , requires all blocks be stacked according to mass.
Revised: The extrapolated transfer task ( p trans ( K ) ), called 4 blocks , requires all blocks be stacked according to mass.

Original: Rewards are given per individual block stacked.
Revised: Rewards are given per individual block stacked.

Original: 5. H IERARCHY AND P RIORS F OR K NOWLEDGE T RANSFER
Revised: 5. H IERARCHY AND P RIORS F OR K NOWLEDGE T RANSFER

Original: Experiments that do not ( (Hier-)RecSAC) / do leverage prior experience, were ran for 10 6 vs 1 .
Revised: Experiments that do not ( (Hier-)RecSAC) / do leverage prior experience, were ran for 10 6 vs 1 .

Original: 5 ∗ 10environment steps.
Revised: 5 ∗ 10environment steps.

Original: APES employs hierarchy, priors and learns m for π H 0 .
Revised: APES employs hierarchy, priors and learns m for π H 0 .

Original: APES- { H20 - S } do not learn m .
Revised: APES- { H20 - S } do not learn m .

Original: The remainder do not employ priors.
Revised: The remainder do not employ priors.

Original: [Table]
Revised: [Table]

Original: Our full setup, APES , leverages hierarchy and priors for skill transfer.
Revised: Our full setup, APES , leverages hierarchy and priors for skill transfer.

Original: The high-level prior is given access to the history (as is common for POMDPs) and learns sparse self-attention m .
Revised: The high-level prior is given access to the history (as is common for POMDPs) and learns sparse self-attention m .

Original: To investigate the importance of priors, we compare against APES-no prior , a baseline from Tirumala et al.
Revised: To investigate the importance of priors, we compare against APES-no prior , a baseline from Tirumala et al.

Original: (2019), with the full APES setup except without a learnt prior.
Revised: (2019), with the full APES setup except without a learnt prior.

Original: Comparing transfer results in Table 2, we see APES ’ drastic gains highlighting the importance of temporal high-level behavioural priors.
Revised: Comparing transfer results in Table 2, we see APES ’ drastic gains highlighting the importance of temporal high-level behavioural priors.

Original: To inspect the transfer importance of the traditional hierarchical setup (with π L ( s t , z t ) ), we compare
Revised: To inspect the transfer importance of the traditional hierarchical setup (with π L ( s t , z t ) ), we compare

Original: APES-no prior against two baselines trained solely on the transfer task.
Revised: APES-no prior against two baselines trained solely on the transfer task.

Original: RecSAC represents a historydependent SAC (Haarnoja et al., 2018d) and Hier-RecSAC a hierarchical equivalent from Wulfmeier et al.
Revised: RecSAC represents a historydependent SAC (Haarnoja et al., 2018d) and Hier-RecSAC a hierarchical equivalent from Wulfmeier et al.

Original: APES-no prior has marginal benefits showing the importance of both hierarchy and priors for transfer.
Revised: APES-no prior has marginal benefits showing the importance of both hierarchy and priors for transfer.

Original: See Table 6 for a detailed explanation of all baseline and ablation setups.
Revised: See Table 6 for a detailed explanation of all baseline and ablation setups.

Original: 5. I NFORMATION A SYMMETRY F OR K NOWLEDGE T RANSFER
Revised: 5. I NFORMATION A SYMMETRY F OR K NOWLEDGE T RANSFER

Original: [Table]
Revised: [Table]

Original: To investigate the importance of IA for transfer, we ablate over high-level priors with increasing levels of asymmetry (each input a subset of the previous): APES- { H20, H10, H1, S } , S denoting an observation-dependent high-level prior, Hi a historydependent one, x t − i : t .
Revised: To investigate the importance of IA for transfer, we ablate over high-level priors with increasing levels of asymmetry (each input a subset of the previous): APES- { H20, H10, H1, S } , S denoting an observation-dependent high-level prior, Hi a historydependent one, x t − i : t .

Original: Crucially, these ablations do not learn m , unlike APES , our full method.
Revised: Crucially, these ablations do not learn m , unlike APES , our full method.

Original: Ablating history lengths is a natural dimension for POMDPs where discovering belief states by history conditioning is crucial (Thrun, 1999).
Revised: Ablating history lengths is a natural dimension for POMDPs where discovering belief states by history conditioning is crucial (Thrun, 1999).

Original: APES-H1, APES-S are hierarchical extensions of Bagatella et al.
Revised: APES-H1, APES-S are hierarchical extensions of Bagatella et al.

Original: ; Galashov et al.
Revised: ; Galashov et al.

Original: (2019) respectively, and APESH20 (representing entire history conditioning) is from Tirumala et al.
Revised: (2019) respectively, and APESH20 (representing entire history conditioning) is from Tirumala et al.

Original: APES-S is also an extension of Pertsch et al.
Revised: APES-S is also an extension of Pertsch et al.

Original: (2020) with π L ( s t , z t ) rather than π L ( z t ) .
Revised: (2020) with π L ( s t , z t ) rather than π L ( z t ) .

Original: Table 2 shows the heavy IA influence, with the trend that conditioning on too little or much information limits performance.
Revised: Table 2 shows the heavy IA influence, with the trend that conditioning on too little or much information limits performance.

Original: The level of influence depends on reward sparsity level: the sparser, the heavier influence, due to rewards guiding exploration less.
Revised: The level of influence depends on reward sparsity level: the sparser, the heavier influence, due to rewards guiding exploration less.

Original: Regardless of the transfer domain being interpolated or extrapolated, IA is influential, suggesting that IA is important over varying levels of sparsity and extrapolation.
Revised: Regardless of the transfer domain being interpolated or extrapolated, IA is influential, suggesting that IA is important over varying levels of sparsity and extrapolation.

Original: 5. T HE E XPRESSIVITY -T RANSFERABILITY T RADE -O FF
Revised: 5. T HE E XPRESSIVITY -T RANSFERABILITY T RADE -O FF

Original: [Figure]
Revised: [Figure]

Original: Transferability Trade-Off .
Revised: Trade-Off .

Original: Distillation loss vs transfer performance increase when pre-training π H .
Revised: Distillation loss vs transfer performance increase when pre-training π H .

Original: The level of information conditioning is shown by marker size.
Revised: The level of information conditioning is shown by marker size.

Original: Absolute transfer performance is shown by marker colour (red - high; blue - low).
Revised: Absolute transfer performance is shown by marker colour (red - high; blue - low).

Original: We fit a second order curve (in red) to show the general expressivity-transferability trend, per Theorems 3.1 and 3.2, that distilling more, by reducing IA, can hurt transfer, due to increased covariate shift.
Revised: We fit a second order curve (in red) to show the general expressivity-transferability trend, per Theorems 3.1 and 3.2, that distilling more, by reducing IA, can hurt transfer, due to increased covariate shift.

Original: See Table 8 for detailed tabular results.
Revised: See Table 8 for detailed tabular results.

Original: To investigate whether Theorems 3.1 and 3.2 are the reason for the apparent expressivitytransferability trade-off seen in
Revised: To investigate whether Theorems 3.1 and 3.2 are the reason for the apparent expressivitytransferability trade-off seen in

Original: Table 2, we plot Figure 4 showing, on the vertical axis, the distillation lossD
Revised: Table 2, we plot Figure 4 showing, on the vertical axis, the distillation loss

Original: [Equation]
Revised: [Equation]

Original: at the end of training over p source ( K ) , verses, on the horizontal axis, the increase in transfer performance (on p trans ( K ) ) when initialising π H as a task agnostic high-level policy pretrained over p source ( K ) (instead of randomly, as is default).
Revised: at the end of training over p source ( K ) , verses, on the horizontal axis, the increase in transfer performance (on p trans ( K ) ) when initialising π H as a task agnostic high-level policy pretrained over p source ( K ) (instead of randomly, as is default).

Original: We ran these additional pre-trained π H experiments to investigate whether covariate shift is the culprit for the degradation in transfer performance when conditioning the highlevel prior π H 0 on additional information.
Revised: We ran these additional pre-trained π H experiments to investigate whether covariate shift is the culprit for the degradation in transfer performance when conditioning the highlevel prior π H 0 on additional information.

Original: By pre-training and transferring π H , we reduce initial trajectory shift and thus initial covariate shift between source and transfer domains (see Section 2.3).
Revised: By pre-training and transferring π H , we reduce initial trajectory shift and thus initial covariate shift between source and transfer domains (see Section 2.2).

Original: This is as no networks are reinitialized during transfer, which would usually lead to an initial shift in policy behaviour across domains.
Revised: This is as no networks are reinitialized during transfer, which would usually lead to an initial shift in policy behaviour across domains.

Original: As per Theorem 3.1, we would expect a larger reduction in covariate shift for the priors that condition on more information.
Revised: As per Theorem 3.1, we would expect a larger reduction in covariate shift for the priors that condition on more information.

Original: If covariate shift were the culprit for reduced transfer performance, we would expect a larger performance gain for those priors conditioned on more more information.
Revised: If covariate shift were the culprit for reduced transfer performance, we would expect a larger performance gain for those priors conditioned on more more information.

Original: Figure 4 demonstrates that is the case in general, regardless of whether the transfer domain is inter- or extra-polated.
Revised: Figure 4 demonstrates that is the case in general, regardless of whether the transfer domain is inter- or extra-polated.

Original: The trend is significantly less apparent for the semi-sparse domain, as here denser rewards guide learning significantly, alleviating covariate shift issues.
Revised: The trend is significantly less apparent for the semi-sparse domain, as here denser rewards guide learning significantly, alleviating covariate shift issues.

Original: We show results for APES- { H20, H10, H1, S } as each input is a subset of the previous.
Revised: We show results for APES- { H20, H10, H1, S } as each input is a subset of the previous.

Original: These relations govern Theorems 3.1 and 3.2.
Revised: These relations govern Theorems 3.1 and 3.2.

Original: Figure 4 and Table 3 show that conditioning on more information improves prior expressivity, reducing distillation losses, as per Theorem 3.2.
Revised: Figure 4 and Table 3 show that conditioning on more information improves prior expressivity, reducing distillation losses, as per Theorem 3.2.

Original: These results together with Table 2, show the impactful expressivity-transferability trade-off of skills, controlled by IA (as per Theorems 3.1 and 3.2), where conditioning on too little or much information limits performance.
Revised: These results together with Table 2, show the impactful expressivity-transferability trade-off of skills, controlled by IA (as per Theorems 3.1 and 3.2), where conditioning on too little or much information limits performance.

Original: 5. APES: A TTENTIVE P RIORS FOR E XPRESSIVE AND T RANSFERABLE S KILLS
Revised: 5. APES: A TTENTIVE P RIORS FOR E XPRESSIVE AND T RANSFERABLE S KILLS

Original: For intra-observation/action attention visualisation, reporting values per individual dimension, see Figure 7.
Revised: For intra-observation/action attention visualisation, reporting values per individual dimension, see Figure 7.

Original: APES learns sparse, domain dependent, attention, primarily focusing on recent actions.
Revised: APES learns sparse, domain dependent, attention, primarily focusing on recent actions.

Original: As seen in Table 2, APES , our full method, strongly outperforms (almost) every baseline and ablation on each transfer domain.
Revised: As seen in Table 2, APES , our full method, strongly outperforms (almost) every baseline and ablation on each transfer domain.

Original: Comparing APES with
Revised: Comparing APES with

Original: APES-H20 , the most comparable approach with the prior fed the same input ( x t − 20: t ), we observe drastic performance gains (of 581% , 183% , 323% for sparse 2 corr , s-sparse 4 corr andblocks , respectively).
Revised: APES-H20 , the most comparable approach with the prior fed the same input ( x t − 20: t ), we observe drastic performance gains.

Original: These results demonstrate the importance of reducing covariate shift (by minimising information conditioning), whilst still supporting expressive behaviours (by exposing the prior to maximal information), only achieved by APES .
Revised: These results demonstrate the importance of reducing covariate shift (by minimising information conditioning), whilst still supporting expressive behaviours (by exposing the prior to maximal information), only achieved by APES .

Original: Table 3 shows H ( m ) , each π H 0 mask’s entropy (a proxy for the amount of information conditioning), vs D KL ( π H
Revised: Table 3 shows H ( m ) , each π H 0 mask’s entropy (a proxy for the amount of information conditioning), vs D KL ( π H

Original: || π H 0 )
Revised: || π H 0 )

Original: (distillation loss), reporting max/min scores across all experiments cycles.
Revised: (distillation loss), reporting max/min scores across all experiments cycles.

Original: We ran 4 random seeds but omit standard deviations as they were negligible.
Revised: We ran 4 random seeds but omit standard deviations as they were negligible.

Original: APES not only attends to minimal information ( H ( m ) ), but for that given level achieves a far lower distillation loss than comparable methods.
Revised: APES not only attends to minimal information ( H ( m ) ), but for that given level achieves a far lower distillation loss than comparable methods.

Original: This demonstrates APES pays attention only where necessary.
Revised: This demonstrates APES pays attention only where necessary.

Original: We inspect APES ’ attention masks m in Figure 5 (aggregated at the observation and action levels).
Revised: We inspect APES ’ attention masks m in Figure 5 (aggregated at the observation and action levels).

Original: Firstly, many attention values tend to 0 (seen clearly in Figure 7) aligning APES closely to Theorems 3.1 and 3.2.
Revised: Firstly, many attention values tend to 0 (seen clearly in Figure 7) aligning APES closely to Theorems 3.1 and 3.2.

Original: Secondly, APES primarily pays attention to the recent history of actions.
Revised: Secondly, APES primarily pays attention to the recent history of actions.

Original: This is interesting, as is inline with recent concurrent work (Bagatella et al., 2022) demonstrating the effectiveness of state-free priors, conditioned on a history of actions, for effective generalization.
Revised: This is interesting, as is inline with recent concurrent work (Bagatella et al., 2022) demonstrating the effectiveness of state-free priors, conditioned on a history of actions, for effective generalization.

Original: Unlike Bagatella et al.
Revised: Unlike Bagatella et al.

Original: (2022) that need exhaustive history lengths sweeps for effective transfer, our approach learns the length in an automated domain dependent manner.
Revised: (2022) that need exhaustive history lengths sweeps for effective transfer, our approach learns the length in an automated domain dependent manner.

Original: As such, our learnt history lengths are distinct for CorridorMaze and Stack .
Revised: As such, our learnt history lengths are distinct for CorridorMaze and Stack .

Original: For CorridorMaze , some attention is paid to the most recent observation s t , which is unsurprising as this information is required to infer how to optimally act whilst at the end of a corridor.
Revised: For CorridorMaze , some attention is paid to the most recent observation s t , which is unsurprising as this information is required to infer how to optimally act whilst at the end of a corridor.

Original: In Figure 7, we plot intra-observation and action attention, and note that for Stack , APES ignores various dimensions of the observation-space, further reducing covariate shift.
Revised: In Figure 7, we plot intra-observation and action attention, and note that for Stack , APES ignores various dimensions of the observation-space, further reducing covariate shift.

Original: Refer to Appendix E.1 for an in-depth analysis of APES ’ full attention maps.
Revised: Refer to Appendix E.1 for an in-depth analysis of APES ’ full attention maps.

Original: 5. H IERARCHY FOR E XPRESSIVITY
Revised: 5. H IERARCHY FOR E XPRESSIVITY

Original: [Table]
Revised: [Table]

Original: To further investigate whether hierarchy is necessary for effective transfer, we compare APES-H1 with APES-H1-flat , which has the same setup except with a flat, non-hierarchical prior (conditioned on x t − 1: t ).
Revised: To further investigate whether hierarchy is necessary for effective transfer, we compare APES-H1 with APES-H1-flat , which has the same setup except with a flat, non-hierarchical prior (conditioned on x t − 1: t ).

Original: With a flat prior, KL-regularization must occur over the raw, rather than latent, action space.
Revised: With a flat prior, KL-regularization must occur over the raw, rather than latent, action space.

Original: Therefore, to adequately compare, we additionally investigate a hierarchical setup where regularization occurs only over the action-space, APES-H1-KL-a .
Revised: Therefore, to adequately compare, we additionally investigate a hierarchical setup where regularization occurs only over the action-space, APES-H1-KL-a .

Original: Transfer results for CorridorMaze are shown in
Revised: Transfer results for CorridorMaze are shown in

Original: Table 4.
Revised: Table 4 ( 7 seeds).

Original: Comparing APES-H1-KL-a and APES-H1-KL-flat , we see the benefits of a hierarchical prior, although less significant for less sparse domains.
Revised: Comparing APES-H1-KL-a and APES-H1-KL-flat , we see the benefits of a hierarchical prior, more significant for sparser domains.

Original: Upon inspection (see Section 5.7), APES-H1-KL-flat is unable to solve the task due to its inability to capture multi-modal behaviours (at corridor intersections).
Revised: Upon inspection (see Section 5.7), APES-H1-KL-flat is unable to solve the task as it cannot capture multi-modal behaviours (at corridor intersections).

Original: Contrasting APES-H1 with APES-H1-KL-a , we see minimal benefits for latent regularization, suggesting with alternate methods for multi-modality, hierarchy may not be necessary.
Revised: Contrasting APES-H1 with APES-H1-KL-a , we see minimal benefits for latent regularization, suggesting with alternate methods for multi-modality, hierarchy may not be necessary.

Original: 5. S KILL -L EVEL E XPLORATION A NALYSIS
Revised: 5. S KILL -L EVEL E XPLORATION A NALYSIS

Original: To gain a further understanding on the effects of hierarchy and priors, we visualise policy rollouts early on during transfer ( 5 ∗ 10 3 steps).
Revised: To gain a further understanding on the effects of hierarchy and priors, we visualise policy rollouts early on during transfer ( 5 ∗ 10 3 steps).

Original: For CorridorMaze (Figure 3), with hierarchy and priors APES explores randomly at the corridor level.
Revised: For CorridorMaze (Figure 3), with hierarchy and priors APES explores randomly at the corridor level.

Original: Hierarchy alone, unable to express preference over high-level skills, leads to temporally uncorrelated behaviours, unable to explore at the corridor level.
Revised: Hierarchy alone, unable to express preference over high-level skills, leads to temporally uncorrelated behaviours, unable to explore at the corridor level.

Original: The flat prior, unable to represent multi-modal behaviours, leads to suboptimal exploration at the intersection of corridors, with the agent often remaining static.
Revised: The flat prior, unable to represent multi-modal behaviours, leads to suboptimal exploration at the intersection of corridors, with the agent often remaining static.

Original: Without priors nor hierarchy, exploration is further hindered, rarely traversing corridor depths.
Revised: Without priors nor hierarchy, exploration is further hindered, rarely traversing corridor depths.

Original: For Stack (Figure 6), APES explores at the block stacking level, alternating block orderings but primarily stacking lighter upon heavier.
Revised: For Stack (Figure 6), APES explores at the block stacking level, alternating block orderings but primarily stacking lighter upon heavier.

Original: Hierarchy alone is unable to stack blocks with temporally uncorrelated skills, exploring at the intra-block stacking level, switching between blocks before successfully stacking, or often interacting with, any individual one.
Revised: Hierarchy alone is unable to stack blocks with temporally uncorrelated skills, exploring at the intra-block stacking level, switching between blocks before successfully stacking, or often interacting with, any individual one.

Original: [Figure]
Revised: [Figure]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Hierarchical frameworks have a long history (Sutton et al., 1999).
Revised: Hierarchical frameworks have a long history (Sutton et al., 1999).

Original: The option RL literature tackle the semi-MDP setup and explorethe benefits that hierarchy and temporal abstraction bring (Nachum et al., 2018; Wulfmeier et al., 2020; Igl et al., 2019; Salter et al., 2022; Kamat & Precup, 2020;
Revised: The options semi-MDP literature explore hierarchy and temporal abstractions (Nachum et al., 2018; Wulfmeier et al., 2020; Igl et al., 2019; Salter et al., 2022; Kamat & Precup, 2020;

Original: Harb et al., 2018; Riemer et al., 2018).
Revised: Harb et al., 2018; Riemer et al., 2018).

Original: Approaches like Wulfmeier et al.
Revised: Approaches like Wulfmeier et al.

Original: (2019; 2020) use hierarchy to enforce knowledge transfer through sharedhierarchical modules.
Revised: (2019; 2020) use hierarchy to enforce knowledge transfer through shared modules.

Original: Gehring et al.
Revised: Gehring et al.

Original: (2021) ensures the hierarchy discovers skills of varying levels for expressivity.
Revised: (2021) use hierarchy to discover skills of varying expressivity levels.

Original: However, for lifelong learning (Parisi et al., 2019; Khetarpal et al., 2020b), where number of skills increase over time, it is unclear how well these approaches will fare, without priors to narrow skill exploration.
Revised: For lifelong learning (Parisi et al., 2019; Khetarpal et al., 2020b), where number of skills increase over time, it is unclear how well these approaches will fare, without priors to narrow skill exploration.

Original: Priors have been used in various fields.
Revised: Priors have been used in various fields.

Original: In the context of offline-RL, Siegel et al.
Revised: In the context of offline-RL, Siegel et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original: (2019) primarily use priors to tackle value overestimation (Levine et al., 2020).
Revised: (2019) primarily use priors to tackle value overestimation (Levine et al., 2020).

Original: In the variational literature, priors have been used to guide latent-space learning (Hausman et al., 2018; Igl et al., 2019;
Revised: In the variational literature, priors have been used to guide latent-space learning (Hausman et al., 2018; Igl et al., 2019;

Original: Pertsch et al., 2020; Merel et al., 2018).
Revised: Pertsch et al., 2020; Merel et al., 2018).

Original: Hausman et al.
Revised: Hausman et al.

Original: (2018) learn episodic skills, limiting their ability to transfer.
Revised: (2018) learn episodic skills, limiting their ability to transfer.

Original: Igl et al.
Revised: Igl et al.

Original: ; Khetarpal et al.
Revised: ; Khetarpal et al.

Original: (2020a) learn options together with priors or interest functions, respectively.
Revised: (2020a) learn options together with priors or interest functions, respectively.

Original: The primer is on-policy, limiting applicability and sample efficiency.
Revised: The primer is on-policy, limiting applicability and sample efficiency.

Original: Both predefine information conditioning of shared modules, limiting transferability.
Revised: Both predefine information conditioning of shared modules, limiting transferability.

Original: Skills and priors have been used in model-based RL to improve planning (Xie et al., 2020; Shi et al., 2022).
Revised: Skills and priors have been used in model-based RL to improve planning (Xie et al., 2020; Shi et al., 2022).

Original: Sikchi et al.
Revised: Sikchi et al.

Original: (2022) use priors to reduce covariate shifts during planning.
Revised: (2022) use priors to reduce covariate shifts during planning.

Original: In contrast, APES ensures the priors themselves experience reduced shifts.
Revised: In contrast, APES ensures the priors themselves experience reduced shifts.

Original: In the multi-task literature, priors have been used to guide exploration (Pertsch et al., 2020; Galashov et al., 2019; Siegel et al., 2020; Pertsch et al., 2021;
Revised: In the multi-task literature, priors have been used to guide exploration (Pertsch et al., 2020; Galashov et al., 2019; Siegel et al., 2020; Pertsch et al., 2021;

Original: Teh et al., 2017), yet without hierarchy expressivity in learnt behaviours is limited.
Revised: Teh et al., 2017), yet without hierarchy expressivity in learnt behaviours is limited.

Original: In the sequential transfer literature, priors have also been used to bias exploration (Pertsch et al., 2020;
Revised: In the sequential transfer literature, priors have also been used to bias exploration (Pertsch et al., 2020;

Original: Ajay et al., 2020; Singh et al., 2020; Bagatella et al., 2022; Goyal et al., 2019;
Revised: Ajay et al., 2020; Singh et al., 2020; Bagatella et al., 2022; Goyal et al., 2019;

Original: Rao et al., 2021; Liu et al., 2022), yet either do not leverage hierarchy (Pertsch et al., 2020) or condition on minimal information (Ajay et al., 2020;
Revised: Rao et al., 2021; Liu et al., 2022), yet either do not leverage hierarchy (Pertsch et al., 2020) or condition on minimal information (Ajay et al., 2020;

Original: Singh et al., 2020;
Revised: Singh et al., 2020;

Original: Rao et al., 2021), limiting expressivity.
Revised: Rao et al., 2021), limiting expressivity.

Original: Unlike APES , Singh et al.
Revised: Unlike APES , Singh et al.

Original: (2020); Bagatella et al.
Revised: (2020); Bagatella et al.

Original: (2022) leverage flow-based transformations to achieve multi-modality.
Revised: (2022) leverage flow-based transformations to achieve multi-modality.

Original: Unlike many previous works, we consider the POMDP setting, arguably more suited for robotics, and learn the information conditioning of priors on based on our expressivity-transferability theorems.
Revised: Unlike many previous works, we consider the POMDP setting, arguably more suited for robotics, and learn the information conditioning of priors on based on our expressivity-transferability theorems.

Original: Whilst most previous works rely on IA, choice is primarily motivated by intuition.
Revised: Whilst most previous works rely on IA, choice is primarily motivated by intuition.

Original: For example,
Revised: For example,

Original: Igl et al.
Revised: Igl et al.

Original: Wulfmeier et al.
Revised: Wulfmeier et al.

Original: (2019; 2020) only employ task or goal asymmetry and Tirumala et al.
Revised: (2019; 2020) only employ task or goal asymmetry and Tirumala et al.

Original: Merel et al.
Revised: Merel et al.

Original: ; Galashov et al.
Revised: ; Galashov et al.

Original: (2019) use exteroceptive asymmetry.
Revised: (2019) use exteroceptive asymmetry.

Original: Salter et al.
Revised: Salter et al.

Original: (2020) investigate a way of learning asymmetry for sim2real domain adaptation, but condition m on observation and state.
Revised: (2020) investigate a way of learning asymmetry for sim2real domain adaptation, but condition m on observation and state.

Original: We consider exploring this direction as future work.
Revised: We consider exploring this direction as future work.

Original: We provide a principled investigation on the role of IA for transfer, proposing a method for automating the choice.
Revised: We provide a principled investigation on the role of IA for transfer, proposing a method for automating the choice.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We employ hierarchical KL-regularized RL to efficiently transfer skills across sequential tasks, showing the effectiveness of combining hierarchy and priors.
Revised: We employ hierarchical KL-regularized RL to efficiently transfer skills across sequential tasks, showing the effectiveness of combining hierarchy and priors.

Original: We theoretically and empirically show the crucial expressivity-transferability trade-off, controlled by IA choice, of skills for hierarchical KLregularized RL.
Revised: We theoretically and empirically show the crucial expressivity-transferability trade-off, controlled by IA choice, of skills for hierarchical KLregularized RL.

Original: Our experiments validate the importance of this trade-off for both interpolated and extrapolated domains.
Revised: Our experiments validate the importance of this trade-off for both interpolated and extrapolated domains.

Original: Given this insight, we introduce APES, ‘ A ttentive P riors for E xpressive and
Revised: Given this insight, we introduce APES, ‘ A ttentive P riors for E xpressive and

Original: Transferable S kills’ automating the IA choice for the high-level prior, by learning it in a data driven, domain dependent, manner.
Revised: Transferable S kills’ automating the IA choice for the high-level prior, by learning it in a data driven, domain dependent, manner.

Original: This is achieved by feeding the entire history to the prior, capturing expressive behaviours, whilst encouraging its attention mask to be low entropic, minimising covariate shift and improving transferability .
Revised: This is achieved by feeding the entire history to the prior, capturing expressive behaviours, whilst encouraging its attention mask to be low entropic, minimising covariate shift and improving transferability .

Original: Experiments over domains of varying sparsity levels demonstrate
Revised: Experiments over domains of varying sparsity levels demonstrate

Original: APES’ consistent superior performance over existing methods, whilst by-passing arduous IA sweeps.
Revised: APES’ consistent superior performance over existing methods, whilst by-passing arduous IA sweeps.

Original: Ablations demonstrate the importance of hierarchy for prior expressivity, by supporting multi-modal behaviours.
Revised: Ablations demonstrate the importance of hierarchy for prior expressivity, by supporting multi-modal behaviours.

Original: Future work will focus on additionally learning the IGFs between hierarchical levels.
Revised: Future work will focus on additionally learning the IGFs between hierarchical levels.

Original: A M ETHOD A. T RAINING R EGIME
Revised: A M ETHOD A. T RAINING R EGIME

Original: In this section, we algorithmically describe our training setup.
Revised: In this section, we algorithmically describe our training setup.

Original: We relate each training phase to the principle equations in the main paper, but note that Appendices A.2 and A.3 outline a more detailed version of these equations that were actually used.
Revised: We relate each training phase to the principle equations in the main paper, but note that Appendices A.2 and A.3 outline a more detailed version of these equations that were actually used.

Original: We note that during BC, we apply DAGGER (Ross et al., 2011), as per Algorithm 2, improving learning rates.
Revised: We note that during BC, we apply DAGGER (Ross et al., 2011), as per Algorithm 2, improving learning rates.

Original: For further details refer to Appendix D.
Revised: For further details refer to Appendix D.

Original: 1: # Full training and transfer regime.
Revised: 1: # Full training and transfer regime.

Original: For BC, gradients are prevented from flowing from π 0 to π .
Revised: For BC, gradients are prevented from flowing from π 0 to π .

Original: In practice π 0 = { π i } i ∈{ 0 ,...,N } , multiple trained priors.
Revised: In practice π 0 = { π i } i ∈{ 0 ,...,N } , multiple trained priors.

Original: During transfer, π H , is reinitialized.
Revised: During transfer, π H , is reinitialized.

Original: 2: 3: # Behavioral Cloning 4: Initialize: policy π , prior π 0 , replay R bc , DAGGER rate r , environment env 5: for Number of BC training steps do 6:
Revised: 2: 3: # Behavioral Cloning 4: Initialize: policy π , prior π 0 , replay R bc , DAGGER rate r , environment env 5: for Number of BC training steps do 6:

Original: R bc , env ← collect( π , R bc , env , True, r ) 7: π , π 0 ← BC update( π , π 0 , R bc ) # Eq.8: end for 9: # Reinforcement Learning 10: Initialize: high level policy π H , critics Q k ∈{ 1 , 2 } , replay R rl , transfer environment env t 11: for Number of RL training steps do 12:
Revised: R bc , env ← collect( π , R bc , env , True, r ) 7: π , π 0 ← BC update( π , π 0 , R bc ) # Eq.8: end for 9: # Reinforcement Learning 10: Initialize: high level policy π H , critics Q k ∈{ 1 , 2 } , replay R rl , transfer environment env t 11: for Number of RL training steps do 12:

Original: R rl , env t ← collect( π , R rl , env t ) 13: π H ← RL policy update( π , π 0 , R rl )
Revised: R rl , env t ← collect( π , R rl , env t ) 13: π H ← RL policy update( π , π 0 , R rl )

Original: # Eq.14:
Revised: # Eq.14:

Original: Q k ← RL critic update( Q k , π , R rl )
Revised: Q k ← RL critic update( Q k , π , R rl )

Original: # Eq.15: end for
Revised: # Eq.15: end for

Original: 1: # Collects experience from either π i or π e , applying DAGGER at a given rate if instructed, and updates R j , env accordingly.
Revised: 1: # Collects experience from either π i or π e , applying DAGGER at a given rate if instructed, and updates R j , env accordingly.

Original: 2: function COLLECT ( π i , R j , env , dag =
Revised: 2: function COLLECT ( π i , R j , env , dag =

Original: False, r = 1 ) 3: x ← env.
Revised: False, r = 1 ) 3: x ← env.

Original: observation () 4: π e ← env.
Revised: observation () 4: π e ← env.

Original: expert () 5: a i ← π i ( x ) 6: a e ← π e ( x ) 7: a ← Bernoulli([ a i , a e ], [ r , 1 - r ]) 8: x ′ , r k , env ← env.
Revised: expert () 5: a i ← π i ( x ) 6: a e ← π e ( x ) 7: a ← Bernoulli([ a i , a e ], [ r , 1 - r ]) 8: x ′ , r k , env ← env.

Original: step ( a ) 9: if dag then 10: a f ← a e 11: else 12: a f ← a i 13: end if 14:
Revised: step ( a ) 9: if dag then 10: a f ← a e 11: else 12: a f ← a i 13: end if 14:

Original: R j ← R j . update ( x , a f , r k , x ′ ) 15: return R j , env 16: end function
Revised: R j ← R j . update ( x , a f , r k , x ′ ) 15: return R j , env 16: end function

Original: A. V ARIATIONAL B EHAVIORAL C LONING AND R EINFORCEMENT L EARNING
Revised: A. V ARIATIONAL B EHAVIORAL C LONING AND R EINFORCEMENT L EARNING

Original: In the following section, we omit APES ’ specific information gating function objective IGF ( x k ) for simplicity and generality.
Revised: In the following section, we omit APES ’ specific information gating function objective IGF ( x k ) for simplicity and generality.

Original: Nevertheless, it is trivial to extend the following derivations to APES .
Revised: Nevertheless, it is trivial to extend the following derivations to APES .

Original: Behavioral Cloning (BC) and KL-Regularized RL, when considered from the variational-inference perspective, share many similarities.
Revised: Behavioral Cloning (BC) and KL-Regularized RL, when considered from the variational-inference perspective, share many similarities.

Original: These similarities become even more apparent when dealing with hierarchical models.
Revised: These similarities become even more apparent when dealing with hierarchical models.

Original: A particularly unifying choice of objective functions for BC and RL that fit with off-policy, generative, hierarchical RL: desirable for sample efficiency, are:
Revised: A particularly unifying choice of objective functions for BC and RL that fit with off-policy, generative, hierarchical RL: desirable for sample efficiency, are:

Original: [Equation]
Revised: [Equation]

Original: L bc , corresponds to the KL-divergence between trajectories from the policy, π , and various priors, π i .
Revised: L bc , corresponds to the KL-divergence between trajectories from the policy, π , and various priors, π i .

Original: For BC, i ∈ { 0 , u, e } , denote the learnt, uniform, and expert priors.
Revised: For BC, i ∈ { 0 , u, e } , denote the learnt, uniform, and expert priors.

Original: For BC, in practice, we train multiple priors in parallel: π 0 =
Revised: For BC, in practice, we train multiple priors in parallel: π 0 =

Original: { π i } i ∈{ 0 ,...,N } .
Revised: { π i } i ∈{ 0 ,...,N } .

Original: We leave this notation out for the remainder of this section for simplicity.
Revised: We leave this notation out for the remainder of this section for simplicity.

Original: When considering only the expert prior, this is the reverse KL-divergence, opposite to what is usually evaluated in the literature, (Pertsch et al., 2020).
Revised: When considering only the expert prior, this is the reverse KL-divergence, opposite to what is usually evaluated in the literature, (Pertsch et al., 2020).

Original: L rl , refers to a lower bound on the expected optimality of each prior log p
Revised: L rl , refers to a lower bound on the expected optimality of each prior log p

Original: π i ( O = 1) ; O denoting the event of achieving maximum return (return referred to as R ( . ) ); refer to (Abdolmaleki et al., 2018), appendix B.4.
Revised: π i ( O = 1) ; O denoting the event of achieving maximum return (return referred to as R ( . ) ); refer to (Abdolmaleki et al., 2018), appendix B.4.

Original: for proof, further explanation, and necessary conditions.
Revised: for proof, further explanation, and necessary conditions.

Original: During transfer using RL, we do not have access to the expert or its demonstrations ( i ⊂ I := i ∈ { 0 , u } ).
Revised: During transfer using RL, we do not have access to the expert or its demonstrations ( i ⊂ I := i ∈ { 0 , u } ).

Original: For hierarchical policies, the KL terms are not easily evaluable.
Revised: For hierarchical policies, the KL terms are not easily evaluable.

Original: [Equation]
Revised: [Equation]

Original: 2 (Tirumala et al., 2019), is a commonly chosen upper bound.
Revised: 2 (Tirumala et al., 2019), is a commonly chosen upper bound.

Original: If sharing modules, e.g. π Li = π L , or using non-hierarchical networks, this bound can be simplified (removing the second or first terms respectively).
Revised: If sharing modules, e.g. π Li = π L , or using non-hierarchical networks, this bound can be simplified (removing the second or first terms respectively).

Original: To make both Eq.
Revised: To make both Eq.

Original: (5) amendable to off-policy training (experience from { π e , π b } , for
Revised: (5) amendable to off-policy training (experience from { π e , π b } , for

Original: BC/RL respectively; π b representing behavioral policy)
Revised: BC/RL respectively; π b representing behavioral policy)

Original: [Equation]
Revised: [Equation]

Original: we introduce importance weighting (IW), removing off-policy bias at the expense of higher variance.
Revised: we introduce importance weighting (IW), removing off-policy bias at the expense of higher variance.

Original: Combining all the above with additional individual term weighting hyperparameters
Revised: Combining all the above with additional individual term weighting hyperparameters

Original: [Equation]
Revised: [Equation]

Original: { β zi
Revised: { β zi

Original: [Equation]
Revised: [Equation]

Original: β ai }, we attain:
Revised: β ai }

Original: [Equation]
Revised: [Equation]

Original: t ν π b [ t ] · r k ( x t , a t ) (cid:35) − (cid:88) i ∈{ 0 ,u } ˜ D π b ( τ ) KL ( π ( τ ) ||
Revised: we attain:

Original: π i ( τ ))
Revised: [Equation]

Original: Where ˜D q ( τ ) KL ( π ( τ ) ||
Revised: Where ˜D q ( τ ) KL ( π ( τ ) ||

Original: π i ( τ ))
Revised: π i ( τ ))

Original: (for { β zi , β ai } = 1 ) is an unbiased estimate for the aforementioned upper bound, using q ’s experience.
Revised: (for { β zi , β ai } = 1 ) is an unbiased estimate for the aforementioned upper bound, using q ’s experience.

Original: ζ ni is the IW for timestep i , between π and arbitrary policy n .
Revised: ζ ni is the IW for timestep i , between π and arbitrary policy n .

Original: ν n [ t ] is the t th element of ν n ; the cumulative IW product at timestep t .
Revised: ν n [ t ] is the t th element of ν n ; the cumulative IW product at timestep t .

Original: Equations 6, 7 are the BC/RL lower bounds used for policy gradients.
Revised: Equations 6, 7 are the BC/RL lower bounds used for policy gradients.

Original: See Appendix B.4 for a derivation, and necessary conditions, of these bounds.
Revised: See Appendix B.4 for a derivation, and necessary conditions, of these bounds.

Original: For BC, this bounds the KL-divergence between hierarchical and expert policies, π, π e .
Revised: For BC, this bounds the KL-divergence between hierarchical and expert policies, π, π e .

Original: For RL, this bounds the expected optimality, for the learnt prior policy, π 0 .
Revised: For RL, this bounds the expected optimality, for the learnt prior policy, π 0 .

Original: Intuitively, maximising this particular bound, maximizes return for both policy and prior, whilst minimizing the disparity between them.
Revised: Intuitively, maximising this particular bound, maximizes return for both policy and prior, whilst minimizing the disparity between them.

Original: Regularising against an uninformative prior, π u , encourages highly-entropic policies, further aiding at exploration and stabilising learning (Igl et al., 2019).
Revised: Regularising against an uninformative prior, π u , encourages highly-entropic policies, further aiding at exploration and stabilising learning (Igl et al., 2019).

Original: In RL, IWs are commonly ignored (Lillicrap et al., 2015; Abdolmaleki et al., 2018; Haarnoja et al., 2018b), thereby considering each sample equally important.
Revised: In RL, IWs are commonly ignored (Lillicrap et al., 2015; Abdolmaleki et al., 2018; Haarnoja et al., 2018b), thereby considering each sample equally important.

Original: This is also convenient for BC, as
Revised: This is also convenient for BC, as

Original: IWs require the expert probability distribution: not usually provided.
Revised: IWs require the expert probability distribution: not usually provided.

Original: We did not observe benefits of using them and therefore ignore them too.
Revised: We did not observe benefits of using them and therefore ignore them too.

Original: We employ module sharing ( π Li =
Revised: We employ module sharing ( π Li =

Original: π L ; unless stated otherwize), and freeze certain modules during distinct phases, and thus never employ more than 2 hyperparameters, β , at any given time, simplifying the hyperparameter optimisation.
Revised: π L ; unless stated otherwize), and freeze certain modules during distinct phases, and thus never employ more than 2 hyperparameters, β , at any given time, simplifying the hyperparameter optimisation.

Original: These weights balance an exploration/exploitation trade-off.
Revised: These weights balance an exploration/exploitation trade-off.

Original: We use a categorical latent space, explicitly marginalising over, rather than using sampling approximations (Jang et al., 2016).
Revised: We use a categorical latent space, explicitly marginalising over, rather than using sampling approximations (Jang et al., 2016).

Original: For BC, we train 
Revised: For BC, we train for

Revised: [Table]

Original: A. C RITIC L EARNING
Revised: A. C RITIC L EARNING

Original: The lower bound presented in Eq.
Revised: The lower bound presented in Eq.

Original: (7) is non-differentiable due to rewards being sampled from the environment.
Revised: (7) is non-differentiable due to rewards being sampled from the environment.

Original: Therefore, as is common in the RL literature (Mnih et al., 2015; Lillicrap et al., 2015), we approximate the return of policy π with a critic, Q .
Revised: Therefore, as is common in the RL literature (Mnih et al., 2015; Lillicrap et al., 2015), we approximate the return of policy π with a critic, Q .

Original: To be sample efficient, we train in an off-policy manner with TD-learning (Sutton, 1988) using the Retrace algorithm (Munos et al., 2016) to provide a low-variance, low-bias, policy evaluation operator:
Revised: To be sample efficient, we train in an off-policy manner with TD-learning (Sutton, 1988) using the Retrace algorithm (Munos et al., 2016) to provide a low-variance, low-bias, policy evaluation operator:

Original: [Equation]
Revised: [Equation]

Original: Where Q ret t represents the policy return evaluated via Retrace.
Revised: Where Q ret t represents the policy return evaluated via Retrace.

Original: Q ′ is the target Q-network, commonly used to stabilize critic learning (Mnih et al., 2015), and is updated periodically with the current Q values.
Revised: Q ′ is the target Q-network, commonly used to stabilize critic learning (Mnih et al., 2015), and is updated periodically with the current Q values.

Original: IWs are not ignored here, and are clipped between [0 , 1] to prevent exploding gradients, (Munos et al., 2016).
Revised: IWs are not ignored here, and are clipped between [0 , 1] to prevent exploding gradients, (Munos et al., 2016).

Original: To further reduce bias and overestimates of our target, Q ret t , we apply the double
Revised: To further reduce bias and overestimates of our target, Q ret t , we apply the double

Original: Q-learning trick, (Hasselt, 2010), and concurrently learn two target Q-networks, Q ′ .
Revised: Q-learning trick, (Hasselt, 2010), and concurrently learn two target Q-networks, Q ′ .

Original: Our critic is trained to minimize the loss in Eq.
Revised: Our critic is trained to minimize the loss in Eq.

Original: (9), which regularizes the critic against the minimum of the two targets produced by both target networks.
Revised: (9), which regularizes the critic against the minimum of the two targets produced by both target networks.

Original: B T HEORY AND D ERIVATIONS
Revised: B T HEORY AND D ERIVATIONS

Original: In this section we provide proofs for the theory introduced in the main paper and in Appendix A.
Revised: In this section we provide proofs for the theory introduced in the main paper and in Appendix A.

Original: B.
Revised: B.

Original: T HEOREM 1.
Revised: T HEOREM 1.

Original: Theorem 1.
Revised: Theorem 1.

Original: The more random variables a network depends on, the larger the covariate shift (input distributional shift, here represented by KL-divergence) encountered across sequential tasks.
Revised: The more random variables a network depends on, the larger the covariate shift (input distributional shift, here represented by KL-divergence) encountered across sequential tasks.

Original: That is, for distributions p , q
Revised: That is, for distributions p , q

Original: D KL ( p ( b ) ∥
Revised: D KL ( p ( b ) ∥

Original: q ( b )) ≥ D KL ( p ( c ) ∥ q ( c )) with b = ( b 0 , b 1 , ..., b n ) and c ⊂ b .
Revised: q ( b )) ≥ D KL ( p ( c ) ∥ q ( c )) with b = ( b 0 , b 1 , ..., b n ) and c ⊂ b .

Original: Proof
Revised: Proof

Original: D KL ( p ( b ) ∥
Revised: D KL ( p ( b ) ∥

Revised: [Equation]

Original: q ( b )) =
Revised: ( b )) =

Original: E p ( b ) (cid:20) log (cid:18) p ( b ) q ( b ) (cid:19)(cid:21) =
Revised: E p ( b ) (cid:20) log (cid:18) p ( b ) q ( b ) (cid:19)(cid:21) =

Original: E p ( d | c ) · p ( c ) (cid:20) log (cid:18) p ( d | c ) · p ( c ) q ( d | c ) · q ( c ) (cid:19)(cid:21) with d ∈ b ⊕ c =
Revised: E p ( d | c ) · p ( c ) (cid:20) log (cid:18) p ( d | c ) · p ( c ) q ( d | c ) · q ( c ) (cid:19)(cid:21) with d ∈ b ⊕ c =

Original: E p ( c ) (cid:20) E p ( d | c ) [1] · log (cid:18) p ( c ) q ( c ) (cid:19)(cid:21) + E p ( c ) (cid:20) E p ( d | c ) (cid:20) log (cid:18) p ( d | c ) q ( d | c ) (cid:19)(cid:21)(cid:21) =
Revised: E p ( c ) (cid:20) E p ( d | c ) [1] · log (cid:18) p ( c ) q ( c ) (cid:19)(cid:21) + E p ( c ) (cid:20) E p ( d | c ) (cid:20) log (cid:18) p ( d | c ) q ( d | c ) (cid:19)(cid:21)(cid:21) =

Original: D KL ( p ( c ) ∥
Revised: D KL ( p ( c ) ∥

Original: q ( c )) + E p ( c ) [D KL ( p ( d | c )
Revised: q ( c )) + E p ( c ) [D KL ( p ( d | c )

Original: ∥ q ( d | c ))] ≥ D KL ( p ( c ) ∥
Revised: ∥ q ( d | c ))] ≥ D KL ( p ( c ) ∥

Original: q ( c )) given
Revised: q ( c )) given

Original: E p ( c ) [D KL ( p ( d | c ) ∥
Revised: E p ( c ) [D KL ( p ( d | c ) ∥

Original: q ( d | c ))] ≥(11)
Revised: q ( d | c ))] ≥(11)

Original: B.
Revised: B.

Original: T HEOREM 2.
Revised: T HEOREM 2.

Original: Theorem 2.
Revised: Theorem 2.

Original: The more random variables a network depends on, the greater its ability to distil knowledge in the expectation (output distributional shift between network and target distribution, here represented by the expected KL-divergence).
Revised: The more random variables a network depends on, the greater its ability to distil knowledge in the expectation (output distributional shift between network and target distribution, here represented by the expected KL-divergence).

Original: That is, for target distribution p and network q with outputs a and possible inputs b , c , d , such that b = ( b 0 , b 1 , ..., b n ) and d ⊂ c ⊂ b
Revised: That is, for target distribution p and network q with outputs a and possible inputs b , c , d , such that b = ( b 0 , b 1 , ..., b n ) and d ⊂ c ⊂ b

Original: E q ( e | d ) [D KL ( p ( a | b )
Revised: E q ( e | d ) [D KL ( p ( a | b )

Original: ∥ q ( a | c ))] ≤ D KL ( p ( a | b )
Revised: ∥ q ( a | c ))] ≤ D KL ( p ( a | b )

Original: ∥ q ( a | d )) with e ∈ d ⊕ c (12)
Revised: ∥ q ( a | d )) with e ∈ d ⊕ c (12)

Original: Proof
Revised: Proof

Original: D KL ( p ( a | b ) ∥
Revised: D KL ( p ( a | b ) ∥

Original: q ( a | d ))
Revised: q ( a | d ))

Original: =
Revised: =

Original: E p ( a | b ) (cid:20) log (cid:18) p ( a | b ) q ( a | d ) (cid:19)(cid:21) =
Revised: E p ( a | b ) (cid:20) log (cid:18) p ( a | b ) q ( a | d ) (cid:19)(cid:21) =

Original: E p ( a | b ) (cid:2) log p ( a | b ) − log E q ( e | d )
Revised: E p ( a | b ) (cid:2) log p ( a | b ) − log E q ( e | d )

Original: [ q ( a | c )] (cid:3) with e ∈ d ⊕ c ≥ E p ( a | b ) · q ( e | d ) (cid:20) log (cid:18) p ( a | b ) q ( a | c ) (cid:19)(cid:21) given Jensen’s Inequality = 
Revised: [ q ( a | c )] (cid:3) with e ∈ d ⊕ c ≥ E p ( a | b ) · q ( e | d ) (cid:20) log (cid:18) p ( a | b ) q ( a | c ) (cid:19)(cid:21) given Jensen’s Inequality = E [Equation] ( e

Original: E q ( e | d ) [D KL 
Revised: [Equation] d ) [D KL (

Original: [Equation]
Revised: [Equation]

Original: B.H IERARCHICAL KL-D IVERGENCE U PPER B OUND
Revised: B.

Revised: H IERARCHICAL KL-D IVERGENCE U PPER B OUND

Original: All proofs in this section ignore multi-task setup for simplicity.
Revised: All proofs in this section ignore multi-task setup for simplicity.

Original: Extending to this scenario is trivial.
Revised: Extending to this scenario is trivial.

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: B. P OLICY G RADIENT L OWER B OUNDS B.4. I MPORTANCE W EIGHTS D ERIVATION
Revised: B. P OLICY G RADIENT L OWER B OUNDS B.4. I MPORTANCE W EIGHTS D ERIVATION

Original: [Equation]
Revised: [Equation]

Original: For β zi , β ai =
Revised: For β zi , β ai =

Original: [Equation]
Revised: [Equation]

Original: where ub (D KL ( π ( τ ) ∥
Revised: where ub (D KL ( π ( τ ) ∥

Original: π i ( τ ))) corresponds to the hierarchical upper bound introduced in Appendix A.
Revised: π i ( τ ))) corresponds to the hierarchical upper bound introduced in Appendix A.

Original: [Equation]
Revised: [Equation]

Original: Proof
Revised: Proof

Original: [Equation]
Revised: [Equation]

Original: π

Original: [Equation]
Revised: [Equation]

Original: ) (cid:13)(cid:13)

Original: [Equation]

Original: [Equation]

Original: ] =

Original: [Equation]
Revised: [Equation]

Original: t

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: B.4. B EHAVIORAL C LONING U PPER B OUND
Revised: B.4. B EHAVIORAL C LONING U PPER B OUND

Original: [Equation]
Revised: [Equation]

Original: Proof
Revised: Proof

Original: [Equation]
Revised: [Equation]

Original: The last line holds true as each weighted term in ˜D q ( τ ) KL ( π ( τ )
Revised: The last line holds true as each weighted term in ˜D q ( τ ) KL ( π ( τ )

Original: ||
Revised: ||

Original: π i ( τ )) corresponds to KL-divergences which are positive.
Revised: π i ( τ )) corresponds to KL-divergences which are positive

Revised: [Equation]

Original: B.4. R EINFORCEMENT L EARNING U PPER B OUND
Revised: B.4. R EINFORCEMENT L EARNING U PPER B OUND

Original: [Equation]
Revised: [Equation]

Original: for β zi , β ai ≥ 1 and r k < [Equation]
Revised: for β zi , β ai ≥ 1 and r k <(20)

Original: Proof
Revised: Proof

Original: [Equation]
Revised: [Equation]

Original: For line 1 proof see (Abdolmaleki et al., 2018).
Revised: For line 1 proof see (Abdolmaleki et al., 2018).

Original: The final 2 lines hold due to positive KL-divergences.
Revised: The final 2 lines hold due to positive KL-divergences.

Original: C E NVIRONMENTS
Revised: C E NVIRONMENTS

Original: Here we cover each environment setup in detail, including the expert setup used for data collection.
Revised: Here we cover each environment setup in detail, including the expert setup used for data collection.

Original: C. C ORRIDOR M AZE
Revised: C. C ORRIDOR M AZE

Original: Intuitively, the agent starts at the intersection of corridors, at the origin, and must traverse corridors, aligned with each dimension of the observation space, in a given ordering.
Revised: Intuitively, the agent starts at the intersection of corridors, at the origin, and must traverse corridors, aligned with each dimension of the observation space, in a given ordering.

Original: This requires the agent to reach the end of the corridor (which we call half-corridor cycle), and return back to the origin, before the corridor is considered complete.
Revised: This requires the agent to reach the end of the corridor (which we call half-corridor cycle), and return back to the origin, before the corridor is considered complete.

Original: s ∈ { 0 , l } c , p ( s 0 ) = 

Revised: s ∈ { 0 , l } c , p ( s 0 ) =

Original: c , k = one-hot task encoding, a ∈ [0 , 1] , r semi-sparse k( x t , a
Revised: c , k = one-hot task encoding, a ∈ [0 , 1] , r semi-sparse k

Original: [Equation]
Revised: [Equation]

Original: = 1 if agent has correctly completed the entire or half-corridor cycle else0 , r sparse k ( x t , a
Revised: if agent has correctly completed the entire or half-corridor cycle else

Original: [Equation]
Revised: [Equation]

Original: 1 if task complete else 0 .
Revised: 1 if task complete else 0 .

Original: Task is considered complete when a desired ordering of corridors have been traversed.
Revised: Task is considered complete when a desired ordering of corridors have been traversed.

Original: c = 5 represents the number of corridors in our experiments.
Revised: c = 5 represents the number of corridors in our experiments.

Original: l = 6 , the lengths of each corridor.
Revised: l = 6 , the lengths of each corridor.

Original:  s j t t [-+
-]
Revised: Observations transition according to deterministic transition function s j t t [-+
-]+1 =

Original: = f ( s j t t , a t ) .
Revised: f ( s j t t , a t ) .

Original: j t corresponds to the index of the current corridor that the agent is in (i.e. j t
Revised: j t corresponds to the index of the current corridor that the agent is in (i.e. j t

Original: =if the agent is in corridor 0 at timestep t ).
Revised: =if the agent is in corridor 0 at timestep t ).

Original: s i corresponds to the i th dimension of the observation.
Revised: s i corresponds to the i th dimension of the observation.

Original: Observations transition incrementally or decrementally down a corridor, and given observation dimension s i , if actions fall into corresponding transition action bins ψ inc , ψ dec .
Revised: Observations transition incrementally or decrementally down a corridor, and given observation dimension s i , if actions fall into corresponding transition action bins ψ inc , ψ dec .

Original: We define the transition function as follows:
Revised: We define the transition function as follows:

Original: [Equation]
Revised: [Equation]

Original: 22) ψ winc ( a t , j ) = bool ( a t in [ j/c, ( j + 0 . 5 · w ) /c ]) , ψ wdec ( a t , j ) = bool ( a t in [ j/c, (2 · j − 0 .
Revised: ψ winc ( a t , j ) = bool ( a t in [ j/c [Equation] ( j [Equation]

Original: 5 · w ) /c ]) .
Revised: 5 · w ) /c ]) , ψ wdec ( a t [Equation] j ) = bool ( a t in [ j/c

Revised: [Equation]

Revised: 2 · j − 0 .

Revised: 5 · w ) /c ]) .

Original: The smaller the w parameter, the narrower the distribution of actions that lead to transitions.
Revised: The smaller the w parameter, the narrower the distribution of actions that lead to transitions.

Original: As such, w , together with r k controls the exploration difficulty of task k .
Revised: As such, w , together with r k controls the exploration difficulty of task k .

Original: We set w = 0 .
Revised: We set w = 0 .

Original: We constrain the observation transitions to not transition outside of the corridor boundaries.
Revised: We constrain the observation transitions to not transition outside of the corridor boundaries.

Original: Furthermore, if the agent is at the origin, s = 

Revised: Furthermore, if the agent is at the origin, s =

Original: c (at the intersection of corridors), then the transition function is ran for all values of j t , thereby allowing the agent to transition into any corridor.
Revised: c (at the intersection of corridors), then the transition function is ran for all values of j t , thereby allowing the agent to transition into any corridor.

Original: C.1. E XPERT S ETUP
Revised: C.1. E XPERT S ETUP

Original: The expert samples actions uniformly within the optimal action bin range, from Eq.
Revised: The expert samples actions uniformly within the optimal action bin range, from Eq.

Original: (22), that leads to the optimal state transition, traversing the correct corridor in the correct direction, according to the task, which corridors have been traversed, and which remain.
Revised: (22), that leads to the optimal state transition, traversing the correct corridor in the correct direction, according to the task, which corridors have been traversed, and which remain.

Original: C. S TACK
Revised: C. S TACK

Original: This domain is adapted from the well known gym robotics FetchPickAndPlace-v0 environment (Plappert et al., 2018).
Revised: This domain is adapted from the well known gym robotics FetchPickAndPlace-v0 environment (Plappert et al., 2018).

Original: The following modifications were made: 1) 3 additional blocks were introduced, with different colours, and a goal pad, 2) object spawn locations were not randomized and were instantiated equidistantly around the goal pad, see Fig. 2, 3) the number of substeps was increased from 20 to 60 , as this reduced episodic lengths, 4) a transparent hollow rectangular tube was placed around the goal pad, to simplify the stacking task and prevent stacked objects from collapsing due to structural instabilities, 5) the arm was always spawned over the goal pad, see figure Fig.
Revised: The following modifications were made: 1) 3 additional blocks were introduced, with different colours, and a goal pad, 2) object spawn locations were not randomized and were instantiated equidistantly around the goal pad, see Fig. 2, 3) the number of substeps was increased from 20 to 60 , as this reduced episodic lengths, 4) a transparent hollow rectangular tube was placed around the goal pad, to simplify the stacking task and prevent stacked objects from collapsing due to structural instabilities, 5) the arm was always spawned over the goal pad, see figure Fig.

Original: 2, 6) the observation space corresponded to gripper position and grasp state, as well as the object positions and relative positions with respect to the arm: velocities were omitted as access to such information may not be realistic for real robotic systems.
Revised: 2, 6) the observation space corresponded to gripper position and grasp state, as well as the object positions and relative positions with respect to the arm: velocities were omitted as access to such information may not be realistic for real robotic systems.

Original: k = one-hot task encoding, r sparse k
Revised: k = one-hot task encoding, r sparse k

Original: ( x t , a t )
Revised: ( x t , a t )

Original: = 1 if correct object has been placed on stack in correct ordering else 0 .
Revised: = 1 if correct object has been placed on stack in correct ordering else 0 .

Original: C.2. E XPERT S ETUP
Revised: C.2. E XPERT S ETUP

Original: The expert is set up to stack the blocks in the given ordering.
Revised: The expert is set up to stack the blocks in the given ordering.

Original: Each individual block stacking cycle consists of six segments: 1) Move the gripper position to a target location 20 cm directly above the block, keeping the gripper open; 2) Vertically lower the gripper to 5 mm over the block, keeping the gripper open; 3)
Revised: Each individual block stacking cycle consists of six segments: 1) Move the gripper position to a target location 20 cm directly above the block, keeping the gripper open; 2) Vertically lower the gripper to 5 mm over the block, keeping the gripper open; 3)

Original: Close the gripper until the object is grasped; 4) Vertically raise the gripper to 20 cm above the initial block position, keeping the gripper closed; 5) Move the gripper to target location 20 cm above the target pad, with the gripper closed; 6) Open the gripper until the object is dropped onto the target pad.
Revised: Close the gripper until the object is grasped; 4) Vertically raise the gripper to 20 cm above the initial block position, keeping the gripper closed; 5) Move the gripper to target location 20 cm above the target pad, with the gripper closed; 6) Open the gripper until the object is dropped onto the target pad.

Original: We use Mujoco’s
Revised: We use Mujoco’s

Original: (Todorov et al., 2012) PD controller, given target relative desired gripper position, which coupled with Mujoco’s inverse kinematics model, produces desired actions.
Revised: (Todorov et al., 2012) PD controller, given target relative desired gripper position, which coupled with Mujoco’s inverse kinematics model, produces desired actions.

Original: We apply gains of 21 to these actions.
Revised: We apply gains of 21 to these actions.

Original: One the target location is reached for a given stage, we proceed to the next.
Revised: One the target location is reached for a given stage, we proceed to the next.

Original: When opening or closing the gripper we apply actions of 0 .
Revised: When opening or closing the gripper we apply actions of 0 .

Original: 05 , − 0 .
Revised: 05 , − 0 .

Original: 1 , for stages 3 and 6.
Revised: 1 , for stages 3 and 6.

Original: For stage 3, we continue closing the gripper until there are contact forces between the gripper and cube.
Revised: For stage 3, we continue closing the gripper until there are contact forces between the gripper and cube.

Original: For stage 6, we continue opening the gripper until the block has dropped such that it is within 14 cm of the target pad.
Revised: For stage 6, we continue opening the gripper until the block has dropped such that it is within 14 cm of the target pad.

Original: To prevent fully deterministic samples, which can be problematic for behavioural cloning, we inject noise into the expert actions.
Revised: To prevent fully deterministic samples, which can be problematic for behavioural cloning, we inject noise into the expert actions.

Original: Specifically, we add Gaussian noise with a diagonal covariance
Revised: Specifically, we add Gaussian noise with a diagonal covariance

Original: [Table] [Table]
Revised: [Table] [Table]

Original: and standard deviation of 0 .
Revised: and standard deviation of 0 .

Original: 2 per dimension.
Revised: 2 per dimension.

Original: We do not apply noise to the gripper closing or opening actions.
Revised: We do not apply noise to the gripper closing or opening actions.

Original: D E XPERIMENTAL S ETUP
Revised: D E XPERIMENTAL S ETUP

Original: We provide the reader with the experimental setup for all training regimes and environments below.
Revised: We provide the reader with the experimental setup for all training regimes and environments below.

Original: We build off the softlearning code base (Haarnoja et al., 2018b).
Revised: We build off the softlearning code base (Haarnoja et al., 2018b).

Original: Algorithmic details not mentioned in the following sections are omitted as are kept constant with the original code base.
Revised: Algorithmic details not mentioned in the following sections are omitted as are kept constant with the original code base.

Original: For all experiments, we sample batch size number of entire episodes of experience during training.
Revised: For all experiments, we sample batch size number of entire episodes of experience during training.

Original: D. M ODEL A RCHITECTURES
Revised: D. M ODEL A RCHITECTURES

Original: We continue by outlining the shared model architectures across domains and experiments.
Revised: We continue by outlining the shared model architectures across domains and experiments.

Original: Each policy network (e.g. π H , π L , π H 0 , π L 0 ) is comprized of a feedforward module outlined in Table 5.
Revised: Each policy network (e.g. π H , π L , π H 0 , π L 0 ) is comprized of a feedforward module outlined in Table 5.

Original: The softlearning repository that we build off (Haarnoja et al., 2018b), applies tanh activation over network outputs, where appropriate, to match the predefined outpute ranges of any given module.
Revised: The softlearning repository that we build off (Haarnoja et al., 2018b), applies tanh activation over network outputs, where appropriate, to match the predefined outpute ranges of any given module.

Original: The critic is also comprized of the same feedforward module, but is not hierarchical.
Revised: The critic is also comprized of the same feedforward module, but is not hierarchical.

Original: To handle historical inputs, we tile the inputs and flatten, to become one large input 1-dimensional array.
Revised: To handle historical inputs, we tile the inputs and flatten, to become one large input 1-dimensional array.

Original: We ensure the input always remains of fixed size by appropriately left padding zeros.
Revised: We ensure the input always remains of fixed size by appropriately left padding zeros.

Original: For π H , π H 

Revised: For π H , π H

Original: we use a categorical latent space of size 10 .
Revised: we use a categorical latent space of size 10 .

Original: We found this dimensionality sufficed for expressing the diverse behaviours exhibited in our domains.
Revised: We found this dimensionality sufficed for expressing the diverse behaviours exhibited in our domains.

Original: Table 6 describes the setup for all the experiments in the main paper, including inputs to each module, level over which KL-regularization occurs ( z or a ), which modules are shared (e.g. π L and π L 0 ), and which modules are reused across sequential tasks.
Revised: Table 6 describes the setup for all the experiments in the main paper, including inputs to each module, level over which KL-regularization occurs ( z or a ), which modules are shared (e.g. π L and π L 0 ), and which modules are reused across sequential tasks.

Original: For the covariate-shift designed experiments in Table 8, we additionally reuse π H (or π L for RecSAC) across domains, and whose input is x t .
Revised: For the covariate-shift designed experiments in Table 8, we additionally reuse π H (or π L for RecSAC) across domains, and whose input is x t .

Original: For all the above experiments, any reused modules are not given access to task-dependent information, namely task-id ( k ) and exteroceptive information (cube locations for Stack domain).
Revised: For all the above experiments, any reused modules are not given access to task-dependent information, namely task-id ( k ) and exteroceptive information (cube locations for Stack domain).

Original: This choice ensures reused modules generalize across task instances.
Revised: This choice ensures reused modules generalize across task instances.

Original: D. B EHAVIOURAL C LONING
Revised: D. B EHAVIOURAL C LONING

Original: For the BC setup, we use a deterministic, noisy, expert controller to create experience to learn off.
Revised: For the BC setup, we use a deterministic, noisy, expert controller to create experience to learn off.

Original: We apply DAGGER (Ross et al., 2011) during data collection and training of policy π as we found this aided at achieving a high success rate at the BC tasks.
Revised: We apply DAGGER (Ross et al., 2011) during data collection and training of policy π as we found this aided at achieving a high success rate at the BC tasks.

Original: Our DAGGER setup intermittently during data collection, with a predefined rate, samples an action from π instead of π e , but still saves BC target action a t as the one that would have been taken by the expert for x k .
Revised: Our DAGGER setup intermittently during data collection, with a predefined rate, samples an action from π instead of π e , but still saves BC target action a t as the one that would have been taken by the expert for x k .

Original: This setup helps mitigate covariate shift during training, between policy and expert.
Revised: This setup helps mitigate covariate shift during training, between policy and expert.

Original: Noise levels were chosen to be small enough so that the expert still succeeded at the task.
Revised: Noise levels were chosen to be small enough so that the expert still succeeded at the task.

Original: We trained our policies for one epoch (once over each collected data sample in the expectation).
Revised: We trained our policies for one epoch (once over each collected data sample in the expectation).

Original: It may be possible to be more sample efficient, by increasing the ratio of gradient steps to data collection, but we did not explore this direction.
Revised: It may be possible to be more sample efficient, by increasing the ratio of gradient steps to data collection, but we did not explore this direction.

Original: The interplay we use between data collection and training over the collected experience, is akin to the RL paradigm.
Revised: The interplay we use between data collection and training over the collected experience, is akin to the RL paradigm.

Original: We build off the softlearning code base (Haarnoja et al., 2018b), so please refer to it for details regarding this interplay
Revised: We build off the softlearning code base (Haarnoja et al., 2018b), so please refer to it for details regarding this interplay

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Refer to Table 7a for BC algorithmic details.
Revised: Refer to Table 7a for BC algorithmic details.

Original: It is important to note here that, although we report five β hyper-parameter values, there are only two degrees of freedom.
Revised: It is important to note here that, although we report five β hyper-parameter values, there are only two degrees of freedom.

Original: As we stop gradients flowing from π 0 to π , choice of β 0 is unimportant (as long as it is not 0 ) as it does not influence the interplay between gradients from individual loss terms.
Revised: As we stop gradients flowing from π 0 to π , choice of β 0 is unimportant (as long as it is not 0 ) as it does not influence the interplay between gradients from individual loss terms.

Original: We set these values to 1 .
Revised: We set these values to 1 .

Original: β ae ’s absolute value is also unimportant, and only its relative value compared to β zu and β au matters.
Revised: β ae ’s absolute value is also unimportant, and only its relative value compared to β zu and β au matters.

Original: We also set β ae to 1 .
Revised: We also set β ae to 1 .

Original: For the remaining two hyper-parameters, β zu , β au , we performed a hyper-parameter sweep over three orders of magnitude, three values across each dimension, to obtain the reported optimal values.
Revised: For the remaining two hyper-parameters, β zu , β au , we performed a hyper-parameter sweep over three orders of magnitude, three values across each dimension, to obtain the reported optimal values.

Original: In practice π 0 = { π i } i ∈{ 0 ,...,N } , multiple trained priors each sharing the same β 0 hyper-parameters.
Revised: In practice π 0 = { π i } i ∈{ 0 ,...,N } , multiple trained priors each sharing the same β 0 hyper-parameters.

Original: For α m , denoting the hyper-paramter in Equation (3) weighing the relative contribution of the π H 0 ’s self-attention entropy objective IGF ( x k ) with relation to the remainder of the RL/BC objectives, we performed a sweep over three ordered of magnitude ( 1 e 0 , 1 e − 1 , 1 e − 2 ).
Revised: For α m , denoting the hyper-paramter in Equation (3) weighing the relative contribution of the π H 0 ’s self-attention entropy objective IGF ( x k ) with relation to the remainder of the RL/BC objectives, we performed a sweep over three ordered of magnitude ( 1 e 0 , 1 e − 1 , 1 e − 2 ).

Original: This sweep was ran independent of all the other sweeps, using the optimal setup for all other hyper-parameters.
Revised: This sweep was ran independent of all the other sweeps, using the optimal setup for all other hyper-parameters.

Original: We chose the hyper-parameter with lowest D KL ( π H ||
Revised: We chose the hyper-parameter with lowest D KL ( π H ||

Original: π H 0 ) , H ( m ) combination.
Revised: π H 0 ) , H ( m ) combination.

Original: Four seeds were ran, as for all experiments.
Revised: Four seeds were ran, as for all experiments.

Original: We observed very small variation in learning across the seeds, and used the best performing seed to bootstrap off for transfer.
Revised: We observed very small variation in learning across the seeds, and used the best performing seed to bootstrap off for transfer.

Original: We separately also performed a hyper-parameter sweep over π learning rate, in the same way as before.
Revised: We separately also performed a hyper-parameter sweep over π learning rate, in the same way as before.

Original: We did not perform a sweep for batch size.
Revised: We did not perform a sweep for batch size.

Original: We found for both BC and RL setups, that conditioning on entire history for π H was not always necessary, and sometimes hurt performance.
Revised: We found for both BC and RL setups, that conditioning on entire history for π H was not always necessary, and sometimes hurt performance.

Original: We state the history lengths used for π H for BC in Table 7a.
Revised: We state the history lengths used for π H for BC in Table 7a.

Original: This value was also used for both π H and Q for the RL setup.
Revised: This value was also used for both π H and Q for the RL setup.

Original: We prevent gradient flow from π 0 to π , to ensure as fair a comparison between ablations as possible: each prior distils knowledge from the same, high performing, policy π and dataset.
Revised: We prevent gradient flow from π 0 to π , to ensure as fair a comparison between ablations as possible: each prior distils knowledge from the same, high performing, policy π and dataset.

Original: If we simultaneously trained multiple π and π 0 pairs (for each distinct prior), it is possible that different learnt priors would influence the quality of each policy π which knowledge is distilled off.
Revised: If we simultaneously trained multiple π and π 0 pairs (for each distinct prior), it is possible that different learnt priors would influence the quality of each policy π which knowledge is distilled off.

Original: In this paper, we are not interested in investigating how priors affect π during BC, but instead how priors influence what knowledge can be distilled and transferred.
Revised: In this paper, we are not interested in investigating how priors affect π during BC, but instead how priors influence what knowledge can be distilled and transferred.

Original: We observed prior KL-distillation loss convergence across tasks and seeds, ensuring a fair comparison.
Revised: We observed prior KL-distillation loss convergence across tasks and seeds, ensuring a fair comparison.

Original: D. R EINFORCEMENT L EARNING
Revised: D. R EINFORCEMENT L EARNING

Original: During this stage of training we freeze the prior and low-level policy (if applicable, depending on the ablation).
Revised: During this stage of training we freeze the prior and low-level policy (if applicable, depending on the ablation).

Original: In general, any reused modules across sequential tasks are frozen (apart from π H for the covariate shift experiments in Table 8).
Revised: In general, any reused modules across sequential tasks are frozen (apart from π H for the covariate shift experiments in Table 8).

Original: Any modules that are not shared (such as π H for most experiments), are initialized randomly across tasks.
Revised: Any modules that are not shared (such as π H for most experiments), are initialized randomly across tasks.

Original: The RL setup is akin to the softlearning repository (Haarnoja et al., 2018b) that we build off.
Revised: The RL setup is akin to the softlearning repository (Haarnoja et al., 2018b) that we build off.

Original: We note any changes in Table 7b.
Revised: We note any changes in Table 7b.

Original: We regularize against the latent or action level for, depending on the ablation, whether or not our models are hierarchical, share low level policies, or use pre-trained modules (low-level policy and prior).
Revised: We regularize against the latent or action level for, depending on the ablation, whether or not our models are hierarchical, share low level policies, or use pre-trained modules (low-level policy and prior).

Original: Therefore, we only ever regularize against, at most, two β hyper-parameters.
Revised: Therefore, we only ever regularize against, at most, two β hyper-parameters.

Original: Hyper-parameter sweeps are performed in the same way as previously.
Revised: Hyper-parameter sweeps are performed in the same way as previously.

Original: We did not sweep over Retrace λ , batch size , or episodic length .
Revised: We did not sweep over Retrace λ , batch size , or episodic length .

Original: For
Revised: For

Original: Retrace, we clip the importance weights between [0 , 1] , like in Munos et al.
Revised: Retrace, we clip the importance weights between [0 , 1] , like in Munos et al.

Original: (2016), and perform λ returns rather than n-step.
Revised: (2016), and perform λ returns rather than n-step.

Original: We found the Retrace operator important for sample-efficient learning.
Revised: We found the Retrace operator important for sample-efficient learning.

Original: E F INAL P OLICY R OLLOUTS AND A NALYSIS E. A TTENTION A NALYSIS
Revised: E F INAL P OLICY R OLLOUTS AND A NALYSIS E. A TTENTION A NALYSIS

Original: We plot the full attention maps for APES in Figure 7, including intra- observation and action attention.
Revised: We plot the full attention maps for APES in Figure 7, including intra- observation and action attention.

Original: For CorridorMaze , attention is primarily paid to the most recent action a t .
Revised: For CorridorMaze , attention is primarily paid to the most recent action a t .

Original: For most observations in the environment (excluding end of the corridor or corridor intersection observations), conditioning on the previous action suffices to infer the optimal next action (e.g. continue traversing the depths of a corridor).
Revised: For most observations in the environment (excluding end of the corridor or corridor intersection observations), conditioning on the previous action suffices to infer the optimal next action (e.g. continue traversing the depths of a corridor).

Original: Therefore, it is understandable that APES has learnt such an attention mechanism.
Revised: Therefore, it is understandable that APES has learnt such an attention mechanism.

Original: For the remainder of the environment observations (such as corridor ends), conditioning on (a history of) observations is necessary to infer optimal behaviour.
Revised: For the remainder of the environment observations (such as corridor ends), conditioning on (a history of) observations is necessary to infer optimal behaviour.

Original: As such APES pays some attention to observations.
Revised: As such APES pays some attention to observations.

Original: For Stack , attention is primarily paid to a short recent history of actions, with the weights decaying further into the past.
Revised: For Stack , attention is primarily paid to a short recent history of actions, with the weights decaying further into the past.

Original: Interestingly, attention over actions corresponding to opening/closing the gripper (the bottom row of Figure 7) decay a lot quicker, suggesting that this information is redundant.
Revised: Interestingly, attention over actions corresponding to opening/closing the gripper (the bottom row of Figure 7) decay a lot quicker, suggesting that this information is redundant.

Original: This makes sense, as there exists strong correlation between gripper actions at successive time-steps, but this correlation decays very quickly.
Revised: This makes sense, as there exists strong correlation between gripper actions at successive time-steps, but this correlation decays very quickly.

Original: Additionally, APES does not pay attention to observations corresponding to gripper position (the final 3 observation rows in Figure 7), as this can be inferred from the remainder of the observation-space as well as the recent history of gripper actions.
Revised: Additionally, APES does not pay attention to observations corresponding to gripper position (the final 3 observation rows in Figure 7), as this can be inferred from the remainder of the observation-space as well as the recent history of gripper actions.

Original: E. C OVARIATE S HIFT A NALYSIS
Revised: E. C OVARIATE S HIFT A NALYSIS

Original: [Table]
Revised: [Table]

Original: In Table 8 we report additional return (over a randomly initialised π H ) achieved by pre-training (over p source ( K ) ) and transferring a task-agnostic high-level policy π H during transfer ( p trans ( K ) ).
Revised: In Table 8 we report additional return (over a randomly initialised π H ) achieved by pre-training (over p source ( K ) ) and transferring a task-agnostic high-level policy π H during transfer ( p trans ( K ) ).

Original: For experiments that are not hierarchical we pre-train an equivalent non-hierarchical agent.
Revised: For experiments that are not hierarchical we pre-train an equivalent non-hierarchical agent.

Original: Theorem 3.1 suggests we would expect a larger improvement in transfer performance for priors that condition on more information.
Revised: Theorem 3.1 suggests we would expect a larger improvement in transfer performance for priors that condition on more information.

Original: Table 8 confirms this trend demonstrating the importance of prior covariate shift in transferability of behaviours.
Revised: Table 8 confirms this trend demonstrating the importance of prior covariate shift in transferability of behaviours.

Original: This trend is less apparent for the semi-sparse domain.
Revised: This trend is less apparent for the semi-sparse domain.

Original: Additionally, for the interpolated transfer task (2 corridor), the solution is entirely in the support of the training set of tasks.
Revised: Additionally, for the interpolated transfer task (2 corridor), the solution is entirely in the support of the training set of tasks.

Original: Na¨ıvely, one would expect pretraining to fully recover lost performance and match the most performant method.
Revised: Na¨ıvely, one would expect pretraining to fully recover lost performance and match the most performant method.

Original: However, this is not the case as the critic, trained solely on the transfer task, quickly encourages sub-optimal out-of-distribution behaviours.
Revised: However, this is not the case as the critic, trained solely on the transfer task, quickly encourages sub-optimal out-of-distribution behaviours.

Original: [Figure]
Revised: [Figure]

Original: E. F INAL P OLICY R OLLOUTS
Revised: E. F INAL P OLICY R OLLOUTS

Original: In this section, we show final policy performance (in terms of episodic rollouts) for APES-H1 , across each transfer domain.
Revised: In this section, we show final policy performance (in terms of episodic rollouts) for APES-H1 , across each transfer domain.

Original: We additionally display the categorical probability distributions for π H ( x k ) and π H 0 ( F ( x k )) across each rollout to analyse the behaviour of each.
Revised: We additionally display the categorical probability distributions for π H ( x k ) and π H 0 ( F ( x k )) across each rollout to analyse the behaviour of each.

Original: F ( . ) denotes the chosen information gating function for the prior (referred to as IGF ( . ) in the main text).
Revised: F ( . ) denotes the chosen information gating function for the prior (referred to as IGF ( . ) in the main text).

Original: For CorridorMaze 2 and 4 corridor , seen in Figs. 8 and 9(a), we see that the full method successfully solves each respective task, correctly traversing the correct ordering of corridors.
Revised: For CorridorMaze 2 and 4 corridor , seen in Figs. 8 and 9(a), we see that the full method successfully solves each respective task, correctly traversing the correct ordering of corridors.

Original: The categorical distributions for these domains remain relatively entropic.
Revised: The categorical distributions for these domains remain relatively entropic.

Original: In general, latent categories cluster into those that lead the agent deeper down a corridor, and those that return the agent to the hallway.
Revised: In general, latent categories cluster into those that lead the agent deeper down a corridor, and those that return the agent to the hallway.

Original: Policy and prior align their categorical distributions in general, as expected.
Revised: Policy and prior align their categorical distributions in general, as expected.

Original: Interestingly, however, the two categorical distributions deviate the most from each-other at the hallway, the bottleneck state (Sutton et al., 1999), where prior multimodality (for hierarchical π 0 ) exists most (e.g. which corridor to traverse next).
Revised: Interestingly, however, the two categorical distributions deviate the most from each-other at the hallway, the bottleneck state (Sutton et al., 1999), where prior multimodality (for hierarchical π 0 ) exists most (e.g. which corridor to traverse next).

Original: In this setting, the policy needs to deviate from the multimodal prior, and traverse only the optimal next corridor.
Revised: In this setting, the policy needs to deviate from the multimodal prior, and traverse only the optimal next corridor.

Original: We also observe, for the hallway, that the prior allocates one category to each of the five corridors.
Revised: We also observe, for the hallway, that the prior allocates one category to each of the five corridors.

Original: Such behaviour would not be possible with a flat prior.
Revised: Such behaviour would not be possible with a flat prior.

Original: Fig. 9(b) plots the same information for Stack 4 blocks .
Revised: Fig. 9(b) plots the same information for Stack 4 blocks .

Original: APES-H1 successfully solves the transfer task, stacking all blocks according to their masses.
Revised: APES-H1 successfully solves the transfer task, stacking all blocks according to their masses.

Original: Similar categorical latent-space trends exist for this domain as the previous.
Revised: Similar categorical latent-space trends exist for this domain as the previous.

Original: Most noteworthy is the behaviour of both policy and prior at the bottleneck state, the location above the block stack, where blocks are placed.
Revised: Most noteworthy is the behaviour of both policy and prior at the bottleneck state, the location above the block stack, where blocks are placed.

Original: This location is visited five times within the episode: at the start s 0 , and four more times upon each stacked block.
Revised: This location is visited five times within the episode: at the start s 0 , and four more times upon each stacked block.

Original: Interestingly, for this state, the prior becomes increasingly less entropic upon each successive visit.
Revised: Interestingly, for this state, the prior becomes increasingly less entropic upon each successive visit.

Original: This suggests that the prior has learnt that the number of feasible high-level actions (corresponding to which block to stack next), reduces upon each visit, as there remains fewer lighter blocks to stack.
Revised: This suggests that the prior has learnt that the number of feasible high-level actions (corresponding to which block to stack next), reduces upon each visit, as there remains fewer lighter blocks to stack.

Original: It is also interesting that for s 0 , the red categorical value is more favoured than the rest.
Revised: It is also interesting that for s 0 , the red categorical value is more favoured than the rest.

Original: Here, the red categorical value corresponds to moving towards cube 0 , the heaviest cube.
Revised: Here, the red categorical value corresponds to moving towards cube 0 , the heaviest cube.

Original: This behaviour is as expected, as during
Revised: This behaviour is as expected, as during

Original: BC, this cube was stacked first more often than the others, given its mass.
Revised: BC, this cube was stacked first more often than the others, given its mass.

Original: For this domain, akin to
Revised: For this domain, akin to

Original: CorridorMaze , the policy deviates most from the prior at the bottleneck state, as here it needs to behave deterministically (regrading which block to stack next).
Revised: CorridorMaze , the policy deviates most from the prior at the bottleneck state, as here it needs to behave deterministically (regrading which block to stack next).

Original: [Figure] rollout
Revised: [Figure] [Figure]