Original: S IMPLE E MERGENT A CTION R EPRESENTATIONS FROM M ULTI -T ASK P OLICY T RAINING
Revised: S IMPLE E MERGENT A CTION R EPRESENTATIONS FROM M ULTI -T ASK P OLICY T RAINING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Low-level sensory and motor signals in the high-dimensional spaces (e.g., image observations or motor torques) in deep reinforcement learning are complicated to understand or harness for downstream tasks directly.
Revised: Low-level sensory and motor signals in the high-dimensional spaces (e.g., image observations or motor torques) in deep reinforcement learning are complicated to understand or harness for downstream tasks directly.

Original: While sensory representations have been widely studied, the representations of actions that form motor skills are yet under exploration.
Revised: While sensory representations have been widely studied, the representations of actions that form motor skills are yet under exploration.

Original: In this work, we find that when a multi-task policy network takes as input states and task embeddings, a space based on the task embeddings emerges to contain meaningful action representations with moderate constraints.
Revised: In this work, we find that when a multi-task policy network takes as input states and task embeddings, a space based on the task embeddings emerges to contain meaningful action representations with moderate constraints.

Original: Within this space, interpolated or composed embeddings can serve as a high-level interface to instruct the agent to perform meaningful action sequences.
Revised: Within this space, interpolated or composed embeddings can serve as a high-level interface to instruct the agent to perform meaningful action sequences.

Original: Empirical results not only show that the proposed action representations have efficacy for intra-action interpolation and inter-action composition with limited or no learning, but also demonstrate their superior ability in task adaptation to strong baselines in Mujoco locomotion tasks.
Revised: Empirical results not only show that the proposed action representations have efficacy for intra-action interpolation and inter-action composition with limited or no learning, but also demonstrate their superior ability in task adaptation to strong baselines in Mujoco locomotion tasks.

Original: The evidence elucidates that learning action representations is a promising direction toward efficient, adaptable, and composable RL, forming the basis of abstract action planning and the understanding of motor signal space.
Revised: The evidence elucidates that learning action representations is a promising direction toward efficient, adaptable, and composable RL, forming the basis of abstract action planning and the understanding of motor signal space.

Original: Anonymous project page: https://sites.google.
Revised: Anonymous project page: https://sites.google.

Original: com/view/emergent-action-representation/

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep reinforcement learning (RL) can learn near-optimal policies to perform low-level actions with pre-defined reward functions.
Revised: Deep reinforcement learning (RL) can learn near-optimal policies to perform low-level actions with pre-defined reward functions.

Original: While the low-level actions are adequate for the specified tasks, it remains challenging to reuse this learned knowledge to accomplish new tasks efficiently.
Revised: While the low-level actions are adequate for the specified tasks, it remains challenging to reuse this learned knowledge to accomplish new tasks efficiently.

Original: By contrast, when humans perform multiple tasks, the low-level muscle movements are naturally summarized into high-level action representations such as “pick up,” or “turn left”, which can be reused in novel tasks with slight modifications.
Revised: By contrast, when humans perform multiple tasks, the low-level muscle movements are naturally summarized into high-level action representations such as “pick up,” or “turn left”, which can be reused in novel tasks with slight modifications.

Original: As a result, though we are aware of our intent to perform a task, we carry out the most complicated movements without thinking about the detailed joint motions or muscle contractions (Kandel et al., 2021).
Revised: As a result, though we are aware of our intent to perform a task, we carry out the most complicated movements without thinking about the detailed joint motions or muscle contractions (Kandel et al., 2021).

Original: By analogy with such abilities of humans, we ask the question: can RL agents have action representations of low-level motor controls, which can be reused, modified, or composed to perform new tasks?
Revised: By analogy with such abilities of humans, we ask the question: can RL agents have action representations of low-level motor controls, which can be reused, modified, or composed to perform new tasks?

Original: As pointed out in Kandel et al.
Revised: As pointed out in Kandel et al.

Original: (2021), “the task of the motor systems is the reverse of the task of the sensory systems.
Revised: (2021), “the task of the motor systems is the reverse of the task of the sensory systems.

Original: Sensory processing generates an internal representation in the brain of the outside world or of the state of the body.
Revised: Sensory processing generates an internal representation in the brain of the outside world or of the state of the body.

Original: Motor processing begins with an internal representation: the desired purpose of movement”.
Revised: Motor processing begins with an internal representation: the desired purpose of movement”.

Original: In the past decade, representation learning has made tremendous progress in representing high-dimensional sensory signals such as images and audios for revealing the geometric and semantic structures hidden in the raw signals (Bengio et al., 2013;
Revised: In the past decade, representation learning has made tremendous progress in representing high-dimensional sensory signals such as images and audios for revealing the geometric and semantic structures hidden in the raw signals (Bengio et al., 2013;

Original: Chen et al., 2018; Kornblith et al., 2019; Chen et al., 2020; Baevski et al., 2020; Radford et al., 2021; Bardes et al., 2021; Bommasani et al., 2021; He et al., 2022).
Revised: Chen et al., 2018; Kornblith et al., 2019; Chen et al., 2020; Baevski et al., 2020; Radford et al., 2021; Bardes et al., 2021; Bommasani et al., 2021; He et al., 2022).

Original: With the help of the generalization ability of sensory representation learning, downstream control tasks can be accomplished efficiently (Nair et al., 2022; Xiao et al., 2022; Yuan et al., 2022).
Revised: With the help of the generalization ability of sensory representation learning, downstream control tasks can be accomplished efficiently (Nair et al., 2022; Xiao et al., 2022; Yuan et al., 2022).

Original: In contrast to the advancement in sensory representation learning, action representation learning is yet under-explored.
Revised: In contrast to the advancement in sensory representation learning, action representation learning is yet under-explored.

Original: Thus, we make a step to study this topic with an aim to find generalizable action representations which can be reused or efficiently adapted to perform new tasks.
Revised: Thus, we make a step to study this topic with an aim to find generalizable action representations which can be reused or efficiently adapted to perform new tasks.

Original: A key idea in sensory representation learning is pretraining with a comprehensive task (or a set of tasks) and reusing the emergent latent representation.
Revised: A key idea in sensory representation learning is pretraining with a comprehensive task (or a set of tasks) and reusing the emergent latent representation.

Original: Inspired by this simple and powerful idea, we train a multi-task policy network, whereas a set of different tasks share the same latent action representation space.
Revised: Inspired by this simple and powerful idea, we train a multi-task policy network, whereas a set of different tasks share the same latent action representation space.

Original: Further, the time-variant sensory and time-invariant ac- tion representations are decoupled and concatenated as the sensory-action representation, which is transformed by a policy network to form the low-level action control.
Revised: Further, the time-variant sensory and time-invariant action representations are decoupled and concatenated as the sensory-action representation, which is transformed by a policy network to form the low-level action control.

Original: Surprisingly, when trained on a comprehensive set of tasks, we discover that such a simple structure learns an emergent selforganized action representation.
Revised: Surprisingly, when trained on a comprehensive set of tasks, we discover that such a simple structure learns an emergent selforganized action representation.

Original: The emergent action representation can be reused for intra-action interpolation, inter-action composition, and efficient task adaptation.
Revised: The emergent action representation can be reused for intra-action interpolation, inter-action composition, and efficient task adaptation.

Original: In particular, we demonstrate the effectiveness of the emergent action representations with Mujoco locomotion environments and show zero-shot composition and few-shot task adaptation in the representation space.
Revised: In particular, we demonstrate the effectiveness of the emergent action representations with Mujoco locomotion environments and show zero-shot composition and few-shot task adaptation in the representation space.

Original: The agents adapt to new tasks in fewer steps than strong meta RL baselines.
Revised: The agents adapt to new tasks in fewer steps than strong meta RL baselines.

Original: Besides the effectiveness of the action representation, we find the decoupled time-variant sensory representation exhibits equivariant properties.
Revised: Besides the effectiveness of the action representation, we find the decoupled time-variant sensory representation exhibits equivariant properties.

Original: The evidence elucidates that reusable and generalizable action representations may lead to efficient, adaptable, and composable RL, thus forming the basis of abstract action planning and understanding motor signal space.
Revised: The evidence elucidates that reusable and generalizable action representations may lead to efficient, adaptable, and composable RL, thus forming the basis of abstract action planning and understanding motor signal space.

Original: The primary contributions in this work are listed as follows:
Revised: The primary contributions in this work are listed as follows:

Original: We put forward the idea of leveraging emergent action representations from multi-task learners to better understand motor action space and accomplish task generalization.
Revised: We put forward the idea of leveraging emergent action representations from multi-task learners to better understand motor action space and accomplish task generalization.

Original: We decouple the state-related and task-related information of the sensory-action representations and reuse them to conduct action planning more efficiently.
Revised: We decouple the state-related and task-related information of the sensory-action representations and reuse them to conduct action planning more efficiently.

Original: Our approach is a strong adapter, which achieves higher rewards with fewer steps than strong meta RL baselines when adapting to new tasks.
Revised: Our approach is a strong adapter, which achieves higher rewards with fewer steps than strong meta RL baselines when adapting to new tasks.

Original: Our approach supports intra-action interpolation as well as inter-action composition by modifying and composing the learned action representations.
Revised: Our approach supports intra-action interpolation as well as inter-action composition by modifying and composing the learned action representations.

Original: Next, we begin our technical discussion right below and leave the discussion of many valuable and related literature to the end.
Revised: Next, we begin our technical discussion right below and leave the discussion of many valuable and related literature to the end.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Soft Actor-Critic.
Revised: Soft Actor-Critic.

Original: In this paper, our approach is built on Soft Actor-Critic (SAC) (Haarnoja et al., 2018).
Revised: In this paper, our approach is built on Soft Actor-Critic (SAC) (Haarnoja et al., 2018).

Original: SAC is a stable off-policy actor-critic algorithm based on the maximum entropy reinforcement learning framework, in which the actor maximizes both the returns and the entropy.
Revised: SAC is a stable off-policy actor-critic algorithm based on the maximum entropy reinforcement learning framework, in which the actor maximizes both the returns and the entropy.

Original: In SAC, there are three types of parameters to update during optimization: the policy parameter ω , the soft
Revised: In SAC, there are three types of parameters to update during optimization: the policy parameter ω , the soft

Original: Q-function parameter θ and a learnable temperature α .
Revised: Q-function parameter θ and a learnable temperature α .

Original: The objectives are:
Revised: The objectives are:

Original: [Equation]
Revised: [Equation]

Original: where H in Equation 1 is a pre-defined minimum expected entropy.
Revised: where H in Equation 1 is a pre-defined minimum expected entropy.

Original: The value function in Equationis parameterized by the weights V ¯ θ in the target Q-network and can be defined by:
Revised: The value function in Equationis parameterized by the weights V ¯ θ in the target Q-network and can be defined by:

Original: [Equation]
Revised: [Equation]

Original: Task Distribution.
Revised: Task Distribution.

Original: We assume the tasks that the agent may meet are drawn from a pre-defined task distribution p ( T ) .
Revised: We assume the tasks that the agent may meet are drawn from a pre-defined task distribution p ( T ) .

Original: Each task in p ( T ) corresponds to a Markov Decision Process (MDP).
Revised: Each task in p ( T ) corresponds to a Markov Decision Process (MDP).

Original: Therefore, a task T can be defined by a tuple ( S , A , p ( s t +1 | s t , a t ) , p ( s 0 ) , R ( s t , a t )) , in which S and A are respectively the state and action space, p ( s t +1 | s t , a t ) the transition probability distribution, p ( s 0 ) the initial state distribution and R ( s t , a t ) the reward function.
Revised: Therefore, a task T can be defined by a tuple ( S , A , p ( s t +1 | s t , a t ) , p ( s 0 ) , R ( s t , a t )) , in which S and A are respectively the state and action space, p ( s t +1 | s t , a t ) the transition probability distribution, p ( s 0 ) the initial state distribution and R ( s t , a t ) the reward function.

Original: The concept of task distribution is frequently employed in meta RL problems, but we have made some modifications and extensions on it to better match with the setting in this work.
Revised: The concept of task distribution is frequently employed in meta RL problems, but we have made some modifications and extensions on it to better match with the setting in this work.

Original: We divide all the task distributions into two main categories, the “uni-modal” task distributions and the “multimodal” task distributions.
Revised: We divide all the task distributions into two main categories, the “uni-modal” task distributions and the “multimodal” task distributions.

Original: Concretely, the two scenarios are defined as follows:
Revised: Concretely, the two scenarios are defined as follows:

Original: • Definition 1 (Uni-modal task distribution):
Revised: • Definition 1 (Uni-modal task distribution):

Original: In a uni-modal task distribution, there is only one modality among all the tasks in the task distribution.
Revised: In a uni-modal task distribution, there is only one modality among all the tasks in the task distribution.

Original: For example, in HalfCheetah-Vel, a Mujoco locomotion environment, we train the agent to run at different target velocities.
Revised: For example, in HalfCheetah-Vel, a Mujoco locomotion environment, we train the agent to run at different target velocities.

Original: Therefore, running is the only modality in this task distribution.
Revised: Therefore, running is the only modality in this task distribution.

Original: • Definition 2 (Multi-modal task distribution):
Revised: • Definition 2 (Multi-modal task distribution):

Original: In contrast to uni-modal task distribution, there are multiple modalities among the tasks in this task distribution.
Revised: In contrast to uni-modal task distribution, there are multiple modalities among the tasks in this task distribution.

Original: A multi-modal task distribution includes tasks of several different uni-modal task distributions.
Revised: A multi-modal task distribution includes tasks of several different uni-modal task distributions.

Original: For instance, we design a multimodal task distribution called HalfCheetah-Run-Jump, which contains two modalities including HalfCheetah BackVel and 
Revised: For instance, we design a multimodal task distribution called HalfCheetah-Run-Jump, which contains two modalities including HalfCheetah-BackVel and HalfCheetah-BackJump.

Original: HalfCheetah

Original: BackJump

Original: The former has been defined in the previous section and the latter contains tasks that train the agent to jump with different reward weight.
Revised: The former has been defined in the previous section, and the latter contains tasks that train the agent to jump with different reward weight.

Original: In our implementation, we actually train four motions in this environment, running, walking, jumping ans standing.
Revised: In our implementation, we actually train four motions in this environment, running, walking, jumping ans standing.

Original: We will leave more details in Section 4 and Appendix A.
Revised: We will leave more details in Section 4 and Appendix A.1.

Original:  E MERGENT A CTION R EPRESENTATIONS FROM M ULTI -T ASK T RAINING
Revised:  E MERGENT A CTION R EPRESENTATIONS FROM M ULTI -T ASK T RAINING

Original: In this section, we first introduce the sensory-task decoupled policy network architecture.
Revised: In this section, we first introduce the sensory-task decoupled policy network architecture.

Original: Next, we discuss the multitask policy training details, along with the additional constraints to the task embedding for the emergence of action representations.
Revised: Next, we discuss the multitask policy training details, along with the additional constraints to the task embedding for the emergence of action representations.

Original: Lastly, we demonstrate the emergence of action representations through various phenomena and applications.
Revised: Lastly, we demonstrate the emergence of action representations through various phenomena and applications.

Original: 3. M ULTITASK P OLICY N ETWORK AND T RAINING
Revised: 3. M ULTITASK P OLICY N ETWORK AND T RAINING

Original: Decoupled embedding and concatenated decoding.
Revised: Decoupled embedding and concatenated decoding.

Original: An abstract high-level task, e.g., “move forward”, typically changes relatively slower than the transient sensory states.
Revised: An abstract high-level task, e.g., “move forward”, typically changes relatively slower than the transient sensory states.

Original: As a simplification, we decouple the latent representation into a time-variant sensory embedding Z s t and a time-invariant task embedding Z T , which is shown in Figure 1.
Revised: As a simplification, we decouple the latent representation into a time-variant sensory embedding Z s t and a time-invariant task embedding Z T , which is shown in Figure 1.

Original: These embeddings concatenate to form a sensoryaction embedding Z A ( s t , T ) =
Revised: These embeddings concatenate to form a sensoryaction embedding Z A ( s t , T ) =

Original: [ Z s t , Z T ] , which is transformed by the policy network (action decoder) ψ to output a low-level action distribution p ( a t ) =
Revised: [ Z s t , Z T ] , which is transformed by the policy network (action decoder) ψ to output a low-level action distribution p ( a t ) =

Original: ψ ( a t |Z s t , Z T ) , e.g., motor torques.
Revised: ψ ( a t |Z s t , Z T ) , e.g., motor torques.

Original: The action decoder ψ is a multi-layer perceptron (MLP) that outputs a Gaussian distribution in the low-level action space A .
Revised: The action decoder ψ is a multi-layer perceptron (MLP) that outputs a Gaussian distribution in the low-level action space A .

Original: Latent sensory embedding (LSE).
Revised: Latent sensory embedding (LSE).

Original: The low-level sensory state information is encoded by an MLP state encoder ϕ into a latent sensory embedding Z s t = ϕ
Revised: The low-level sensory state information is encoded by an MLP state encoder ϕ into a latent sensory embedding Z s t = ϕ

Original: ( s t ) ∈ R m .
Revised: ( s t ) ∈ R m .

Original: It includes the proprioceptive information of each time step.
Revised: It includes the proprioceptive information of each time step.

Original: LSE is time-variant in an RL trajectory, and the state encoder is shared among different tasks.
Revised: LSE is time-variant in an RL trajectory, and the state encoder is shared among different tasks.

Original: We use LSE and sensory representation interchangeably in this paper.
Revised: We use LSE and sensory representation interchangeably in this paper.

Original: Latent task embedding (LTE).
Revised: Latent task embedding (LTE).

Original: A latent task embedding Z T ∈ R d encodes the time-invariant knowledge of a specific task.
Revised: A latent task embedding Z T ∈ R d encodes the time-invariant knowledge of a specific task.

Original: Let’s assume we are going to train N different tasks, and their embeddings form an LTE set {Z T N } .
Revised: Let’s assume we are going to train N different tasks, and their embeddings form an LTE set {Z T N } .

Original: These N different tasks share the same state encoder ϕ and action decoder ψ ; in other words, these N tasks share the same policy network interface, except for their task embeddings being different.
Revised: These N different tasks share the same state encoder ϕ and action decoder ψ ; in other words, these N tasks share the same policy network interface, except for their task embeddings being different.

Original: For implementation, we adopt a linear encoder, which takes as input the one-hot encodings of different training tasks, to initialize the set {Z T N } .
Revised: For implementation, we adopt a linear encoder, which takes as input the one-hot encodings of different training tasks, to initialize the set {Z T N } .

Original: This task encoder is learnable during training.
Revised: This task encoder is learnable during training.

Original: After training, the LTE interface can be reused as a high-level action interface.
Revised: After training, the LTE interface can be reused as a high-level action interface.

Original: Hence, we use LTE and action representation interchangeably in this paper.
Revised: Hence, we use LTE and action representation interchangeably in this paper.

Original: Training of the multi-task policy networks.
Revised: Training of the multi-task policy networks.

Original: A detailed description of the multi-task training is demonstrated in Algorithm 1.
Revised: A detailed description of the multi-task training is demonstrated in Algorithm 1.

Original: When computing objectives and their gradients, we still use policy π parameterized by ω to indicate all the parameters in the state encoder, task encoder and the action decoder.
Revised: When computing objectives and their gradients, we still use policy π parameterized by ω to indicate all the parameters in the state encoder, task encoder and the action decoder.

Original: The overall training procedure is based on SAC.
Revised: The overall training procedure is based on SAC.

Original: The only difference is that the policy network and Q networks additionally take as input the LTE Z T and a one-hot task label, respectively.
Revised: The only difference is that the policy network and Q networks additionally take as input the LTE Z T and a one-hot task label, respectively.

Original: During training, we also apply two techniques to constrain this space: 1) we normalize the LTEs so that they lie on a hypersphere; 2) we inject a random noise to the LTEs to enhance the smoothness of the space.
Revised: During training, we also apply two techniques to constrain this space: 1) we normalize the LTEs so that they lie on a hypersphere; 2) we inject a random noise to the LTEs to enhance the smoothness of the space.

Original: Algorithm
Revised: Algorithm

Original: Input: Training task set {T N } ∼ p
Revised: Input: Training task set {T N } ∼ p

Original: ( T ) ,
Revised: ( T ) ,

Original: θ 1 , θ 2 , ω θ 1 ← θ 1 , θ 2 ← θ 2 , B ← ∅  for each pre-train epoch do for T i in {T n } do
Revised: θ 1 , θ 2 , ω θ 1 ← θ 1 , θ 2 ← θ 2 , B ← ∅ Initialize LTE set {Z T N } for {T N } for each pre-train epoch do for T i in {T n } do

Original: Sample a batch B i of multi-task RL transitions with π ω
Revised: Sample a batch B i of multi-task RL transitions with π ω

Original: B ← B ∪ B i end for end for for each train epoch do
Revised: B ← B ∪ B i end for end for for each train epoch do

Original: Sample RL batch b ∼ B for all transition data in b do
Revised: Sample RL batch b ∼ B for all transition data in b do

Original: Z s t = ϕ
Revised: Z s t = ϕ

Original: ( s t ) (cid:101) Z T i
Revised: ( s t ) (cid:101) Z T i

Original: = normalize ( Z T i + n ) and n ∼ N (0 , σ 2 )
Revised: = normalize ( Z T i + n ) and n ∼ N (0 , σ 2 )

Original: Sample action a t ∼ ψ ( ·|Z s t , (cid:101) Z T i ) for computing SAC objectives end for for each optimization step do
Revised: Sample action a t ∼ ψ ( ·|Z s t , (cid:101) Z T i ) for computing SAC objectives end for for each optimization step do

Original: Compute SAC objectives J ( α ) , J π ( ω ) , J Q ( θ ) with b based on Equation
Revised: Compute SAC objectives J ( α ) , J π ( ω ) , J Q ( θ ) with b based on Equation

Original: Update SAC parameters end for end for
Revised: Update SAC parameters end for end for

Original: Output: The optimal model of state encoder ϕ ∗ and action decoder ψ ∗ and a set of LTEs {Z T N }
Revised: Output: The optimal model of state encoder ϕ ∗ and action decoder ψ ∗ and a set of LTEs {Z T N }

Original:  T HE E MERGENCE OF A CTION R EPRESENTATION
Revised: 3. T HE E MERGENCE OF A CTION R EPRESENTATION

Original: After we train the multi-task policy network with a comprehensive set of tasks, where the LTE vectors in {Z T N } share the same embedding space, we find that {Z T N } self-organizes into a geometrically and semantically meaningful structure.
Revised: After we train the multi-task policy network with a comprehensive set of tasks, where the LTE vectors in {Z T N } share the same embedding space, we find that {Z T N } self-organizes into a geometrically and semantically meaningful structure.

Original: Tasks with the same modality are embedded in a continuous fashion, which facilitates intra-task interpolation.
Revised: Tasks with the same modality are embedded in a continuous fashion, which facilitates intra-task interpolation.

Original: Surprisingly, the composition of task embeddings from different modalities leads to novel tasks, e.g., “run” + “jump” =
Revised: Surprisingly, the composition of task embeddings from different modalities leads to novel tasks, e.g., “run” + “jump” =

Original: “jump run”.
Revised: “jump run”.

Original: Further, the action representation can be used for efficient task adaptation.
Revised: Further, the action representation can be used for efficient task adaptation.

Original: Visualization also reveals interesting geometric structures in task embedding and sensory representation spaces.
Revised: Visualization also reveals interesting geometric structures in task embedding and sensory representation spaces.

Original: In this subsection, we dive into these intriguing phenomena, demonstrating the emergence of action representation and showing the generalization of the emergent action representation.
Revised: In this subsection, we dive into these intriguing phenomena, demonstrating the emergence of action representation and showing the generalization of the emergent action representation.

Original: Task interpolation & composition.
Revised: Task interpolation & composition.

Original: After training the RL agent to accomplish multiple tasks, we select two pre-trained tasks and generate a new LTE through linear integration between the LTEs of the two chosen tasks.
Revised: After training the RL agent to accomplish multiple tasks, we select two pre-trained tasks and generate a new LTE through linear integration between the LTEs of the two chosen tasks.

Original: The newly-generated task embedding is expected to conduct the agent to perform another different task.
Revised: The newly-generated task embedding is expected to conduct the agent to perform another different task.

Original: The generated LTE is defined by:
Revised: The generated LTE is defined by:

Original: [Equation]
Revised: [Equation]

Original: where i, j are the indices of the selected tasks and Z T i , Z T j are their corresponding LTEs.
Revised: where i, j are the indices of the selected tasks and Z T i , Z T j are their corresponding LTEs.

Original: β is a hyperparameter ranging in (0,1).
Revised: β is a hyperparameter ranging in (0,1).

Original: The function f ( · ) is a regularization function related to the pre- defined quality of the LTE Space.
Revised: The function f ( · ) is a regularization function related to the pre- defined quality of the LTE Space.

Original: For instance, in this paper, f ( · ) is a normalization function to extend or shorten the result of interpolation to a unit sphere.
Revised: For instance, in this paper, f ( · ) is a normalization function to extend or shorten the result of interpolation to a unit sphere.

Original: A new task is interpolated by applying the aforementioned operation on the LTEs of tasks sampled from a uni-modal distribution.
Revised: A new task is interpolated by applying the aforementioned operation on the LTEs of tasks sampled from a uni-modal distribution.

Original: The interpolated task usually has the same semantic meaning as the source tasks while having different quantity in specific parameters, e.g., running with different speeds.
Revised: The interpolated task usually has the same semantic meaning as the source tasks while having different quantity in specific parameters, e.g., running with different speeds.

Original: A new task is composed by applying the same operation on tasks sampled from a multimodal distribution.
Revised: A new task is composed by applying the same operation on tasks sampled from a multimodal distribution.

Original: The newly composed task usually lies in a new modality between the source tasks.
Revised: The newly composed task usually lies in a new modality between the source tasks.

Original: For example, when we compose “run” and “jump” together, we will have a combination of an agent running while trying to jump.
Revised: For example, when we compose “run” and “jump” together, we will have a combination of an agent running while trying to jump.

Original: Efficient adaptation.
Revised: Efficient adaptation.

Original: We find that an agent trained with the multi-task policy network can adapt to unseen tasks quickly by only optimizing the LTEs.
Revised: We find that an agent trained with the multi-task policy network can adapt to unseen tasks quickly by only optimizing the LTEs.

Original: This shows that the LTEs learn a general pattern of the overall task distribution.
Revised: This shows that the LTEs learn a general pattern of the overall task distribution.

Original: When given a new task after pre-training, the agent explores in the
Revised: When given a new task after pre-training, the agent explores in the

Original: LTE Space to find a suitable LTE for the task.
Revised: LTE Space to find a suitable LTE for the task.

Original: Specifically, we perform a gradient-free cross-entropy method (CEM) (De Boer et al., 2005) in the LTE space for accomplishing the desired task.
Revised: Specifically, we perform a gradient-free cross-entropy method (CEM) (De Boer et al., 2005) in the LTE space for accomplishing the desired task.

Original: Detailed description can be found in Algorithm 2.
Revised: Detailed description can be found in Algorithm 2.

Original: Geometric Structures of LTEs and the LSEs.
Revised: Geometric Structures of LTEs and the LSEs.

Original: We visualize the LSEs and LTEs to understand their geometric structures.
Revised: We visualize the LSEs and LTEs to understand their geometric structures.

Original: The results are deferred to Section 4.5.
Revised: The results are deferred to Section 4.5.

Original: Algorithm 
Revised: Algorithm 2 Adaptation via LTE

Original: Input: Adaptation task T ∼ p ( T ) , ϕ ∗ , ψ ∗ , capacity of elite set m , number of sampling n
Revised: Input: Adaptation task T ∼ p ( T ) , ϕ ∗ , ψ ∗ , capacity of elite set m , number of sampling n

Original: Initialize the elite set Z e with m randomly sampled LTEs from the LTE Space for each adapt epoch do
Revised: Initialize the elite set Z e with m randomly sampled LTEs from the LTE Space for each adapt epoch do

Original: Initialize the overall test set by Z ← ∅ for Z i in Z e do
Revised: Initialize the overall test set by Z ← ∅ for Z i in Z e do

Original: Sample n LTEs
Revised: Sample n LTEs

Original: Z
Revised: Z

Original: i 1 , . . .
Revised: i 1 , . . .

Original: , Z in near Z i Z ← Z ∪ {Z i , Z i 1 , . . .
Revised: , Z in near Z i Z ← Z ∪ {Z i , Z i 1 , . . .

Original: , Z in } end for for Z j in Z do while not done doZ
Revised: , Z in } end for for Z j in Z do while not done do

Original: [Equation]
Revised: [Equation]

Original: t = ϕ

Original: [Equation]

Original: ( s t ) , a t ∼ ψ

Original: [Equation]

Original: [Equation]

Original:  EXPERIMENTS
Revised:  EXPERIMENTS

Original: In this section, we first demonstrate the training process and performance of the multi-task policy network.
Revised: In this section, we first demonstrate the training process and performance of the multi-task policy network.

Original: Then, we use the LTEs as a high-level action interface to instruct the agents to perform unseen skills through interpolation without any training.
Revised: Then, we use the LTEs as a high-level action interface to instruct the agents to perform unseen skills through interpolation without any training.

Original: After that, we conduct experiments to evaluate the effectiveness of the LTEs in task adaptation.
Revised: After that, we conduct experiments to evaluate the effectiveness of the LTEs in task adaptation.

Original: Lastly, we visualize the LSEs and LTEs to further understand the structure of the state and action representation.
Revised: Lastly, we visualize the LSEs and LTEs to further understand the structure of the state and action representation.

Original: We use e mergent a ction r epresentation (EAR) to refer to the policy using the LTEs.
Revised: We use e mergent a ction r epresentation (EAR) to refer to the policy using the LTEs.

Original: 4. EXPERIMENTAL SETUPS
Revised: 4. EXPERIMENTAL SETUPS

Original: Environments.
Revised: Environments.

Original: We evaluate our method on five continuous control environments (HalfCheetahVel, Ant-Dir, Hopper-Vel, Walker-Vel, HalfCheetah-Run-Jump) based on OpenAI Gym and the
Revised: We evaluate our method on five continuous control environments (HalfCheetahVel, Ant-Dir, Hopper-Vel, Walker-Vel, HalfCheetah-Run-Jump) based on OpenAI Gym and the

Original: Mujoco simulator.
Revised: Mujoco simulator.

Original: Detailed descriptions of these RL benchmarks are listed in Appendix A.1.
Revised: Detailed descriptions of these RL benchmarks are listed in Appendix A.1.

Original: Baselines.
Revised: Baselines.

Original: We compare EAR-SAC, the emergent action representation based SAC with several multi-task RL and meta RL baselines.
Revised: We compare EAR-SAC, the emergent action representation based SAC with several multi-task RL and meta RL baselines.

Original: For multi-task RL baselines, we use multi-head multi-task
Revised: For multi-task RL baselines, we use multi-head multi-task

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: deviation is calculated among 3 seeds.

Original: SAC (MHMT-SAC) and one-hot embedding SAC (OHE-SAC; for ablation).
Revised: -SAC) and one-hot embedding SAC (OHE-SAC; for ablation).

Original: For meta RL baselines, we use MAML (Finn et al., 2017) and PEARL (Rakelly et al., 2019).
Revised: For meta RL baselines, we use MAML (Finn et al., 2017) and PEARL (Rakelly et al., 2019).

Original: Detailed descriptions of these baselines are listed in Appendix A.2.
Revised: Detailed descriptions of these baselines are listed in Appendix A.2.

Original: 4. M ULTI - TASK T RAINING FOR A CTION R EPRESENTATIONS
Revised: 4. M ULTI - TASK T RAINING FOR A CTION R EPRESENTATIONS

Original: In this section, we train the multi-task network and evaluate whether sensory-action representations can boost training efficiency.
Revised: In this section, we train the multi-task network and evaluate whether sensory-action representations can boost training efficiency.

Original: EAR-SAC is compared with all the multi-task RL and meta RL baselines on final rewards in Table 1 and additionally compared with off-policy baselines on the training efficiency in Figure 2.
Revised: EAR-SAC is compared with all the multi-task RL and meta RL baselines on final rewards in Table 1 and additionally compared with off-policy baselines on the training efficiency in Figure 2.

Original: We find that EAR-SAC outperforms all the baselines in terms of training efficiency.
Revised: We find that EAR-SAC outperforms all the baselines in terms of training efficiency.

Original: In environments with high-dimensional observations (e.g., Ant-Dir), EAR-SAC achieves large performance advantage against the baselines.
Revised: In environments with high-dimensional observations (e.g., Ant-Dir), EAR-SAC achieves large performance advantage against the baselines.

Original: We attribute this to that the learned action representation space may provide meaningful priors when the policy is trained with multiple tasks.
Revised: We attribute this to that the learned action representation space may provide meaningful priors when the policy is trained with multiple tasks.

Original: 4. A CTION R EPRESENTATION AS A H IGH - LEVEL C ONTROL I NTERFACE
Revised: 4. A CTION R EPRESENTATION AS A H IGH - LEVEL C ONTROL I NTERFACE

Original: [Figure]
Revised: 2m/s 1.8m/s, β=0. 1m/s 1.5m/s, β=0. 1.2m/s, β=0. Direction

Original: In this section, we control the agent by recomposing the
Revised: In this section, we control the agent by recomposing the

Original: LTEs.
Revised: LTEs.

Original: We perform intra-action interpolation and interaction composition in uni-modal task distributions and multi-modal task distributions respectively.
Revised: We perform intra-action interpolation and interaction composition in uni-modal task distributions and multi-modal task distributions respectively.

Original: Intra-action interpolation.
Revised: Intra-action interpolation.

Original: Intra-action interpolation is conducted in HalfCheetah-Vel, Ant-Dir, Hopper-Vel, and Walker-Vel.
Revised: Intra-action interpolation is conducted in HalfCheetah-Vel, Ant-Dir, Hopper-Vel, and Walker-Vel.

Original: We interpolate the action representations between two tasks using Equation 5.
Revised: We interpolate the action representations between two tasks using Equation 5.

Original: The coefficient β is searched to better fit the target task.
Revised: The coefficient β is searched to better fit the target task.

Original: An interpolation example in HalfCheetah-Vel is demonstrated in Figure 
Revised: An interpolation example in HalfCheetah-Vel is demonstrated in Figure 3.

Original: We select the tasks of running at 1 m/s and 2 m/s to be interpolated and get three interpolated tasks: run at 1.2 m/s, 1.5 m/s, 1.7 m/s. We perform evaluation on each task for a trajectory, and visualize them.
Revised: We select the tasks of running at 1 m/s and 2 m/s to be interpolated and get three interpolated tasks: run at 1.2 m/s, 1.5 m/s, 1.7 m/s. We perform evaluation on each task for a trajectory, and visualize them.

Original: In each task, we make the agent start from the same point (not plotted in the figure) and terminate at 100 time steps.
Revised: In each task, we make the agent start from the same point (not plotted in the figure) and terminate at 100 time steps.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Only part of the whole scene is visualized in the figure and we mark the terminals with red flags.
Revised: Only part of the whole scene is visualized in the figure and we mark the terminals with red flags.

Original: We find that through task interpolation, the agent manages to accomplish these interpolated tasks without any training.
Revised: We find that through task interpolation, the agent manages to accomplish these interpolated tasks without any training.

Original: We leave other results in all environments to Appendix A.4.
Revised: We leave other results in all environments to Appendix A.4.

Original: Inter-action composition.
Revised: Inter-action composition.

Original: Inter-action composition is conducted in HalfCheetah-Run-Jump, in which we merge two uni-modal task distributions.
Revised: Inter-action composition is conducted in HalfCheetah-Run-Jump, in which we merge two uni-modal task distributions.

Original: Taking HalfCheetah-BackVel and HalfCheetahBackJump as an example, we find that the agent has learned four motions: walking backward, running backward, standing with its front leg
Revised: Taking HalfCheetah-BackVel and HalfCheetahBackJump as an example, we find that the agent has learned four motions: walking backward, running backward, standing with its front leg

Original: [Figure]
Revised: [Figure]

Original: and jumping with its front leg.
Revised: and jumping with its front leg.

Original: We select walkstanding as a composition task pair and run-jumping as the other and compose them with Equation 5.
Revised: We select walkstanding as a composition task pair and run-jumping as the other and compose them with Equation 5.

Original: These two generated representations are evaluated in 1000 time steps and part of their evaluation trajectories are visualized in Figure 4.
Revised: These two generated representations are evaluated in 1000 time steps and part of their evaluation trajectories are visualized in Figure 4.

Original: We find that the walk-standing representation enables the agent to walk backward with a standing posture despite some pauses to keep itself balance, while the run-jumping representation helps the agent to jump when running backward after some trials.
Revised: We find that the walk-standing representation enables the agent to walk backward with a standing posture despite some pauses to keep itself balance, while the run-jumping representation helps the agent to jump when running backward after some trials.

Original: These empirical results indicate that the LTEs can be used as action representations by composing them for different new tasks.
Revised: These empirical results indicate that the LTEs can be used as action representations by composing them for different new tasks.

Original: 4. T ASK A DAPTATION WITH A CTION R EPRESENTATIONS
Revised: 4. T ASK A DAPTATION WITH A CTION R EPRESENTATIONS

Original: In this section, we assess how well an agent can adapt to new tasks by only updating the LTEs.
Revised: In this section, we assess how well an agent can adapt to new tasks by only updating the LTEs.

Original: We compare the agent’s adaptation ability with the meta RL baselines (MAML,PEARL) and the ablation baseline (OHE-SAC).
Revised: We compare the agent’s adaptation ability with the meta RL baselines (MAML,PEARL) and the ablation baseline (OHE-SAC).

Original: The results in Figure 5 demonstrate that EAR can adapt to new tasks and achieve to the converged performance in no more than three epochs, outperforming the baseline meta RL methods.
Revised: The results in Figure 5 demonstrate that EAR can adapt to new tasks and achieve to the converged performance in no more than three epochs, outperforming the baseline meta RL methods.

Original: We note that, with the help of the LTE space, we can adapt to new tasks using zero-th order optimization method.
Revised: We note that, with the help of the LTE space, we can adapt to new tasks using zero-th order optimization method.

Original: 4. V ISUALIZATION OF S TATE AND A CTION R EPRESENTATIONS
Revised: 4. V ISUALIZATION OF S TATE AND A CTION R EPRESENTATIONS

Original: In this section, we further analyze and visualize the sensory-action representations based on the
Revised: In this section, we further analyze and visualize the sensory-action representations based on the

Original: HalfCheetah environment.
Revised: HalfCheetah environment.

Original: [Figure]
Revised: [Figure]

Original: representations (right) in HalfCheetah-Vel.

Original: To make the period of running motion easier to observe, we reduce the time interval between adjacent time steps.
Revised: To make the period of running motion easier to observe, we reduce the time interval between adjacent time steps.

Original: For now a period of the motion is 15 steps, while in the original setting it is only 6 steps.
Revised: For now a period of the motion is 15 steps, while in the original setting it is only 6 steps.

Original: [Figure]
Revised: [Figure]

Original: The sensory representations.
Revised: The sensory representations.

Original: In HalfCheetah-Vel, the halfcheetah agent is encouraged to run at a specific velocity, thus making its motion periodical.
Revised: In HalfCheetah-Vel, the halfcheetah agent is encouraged to run at a specific velocity, thus making its motion periodical.

Original: Therefore, the ideal raw signals and the
Revised: Therefore, the ideal raw signals and the

Original: LSEs in the trajectory should be periodical as well.
Revised: LSEs in the trajectory should be periodical as well.

Original: After conducting Principal Components Analysis (PCA) (Abdi & Williams, 2010) to reduce the dimension of all the raw signals and the sensory representations collected in a trajectory, we visualize them for different time steps in Figure 6.
Revised: After conducting Principal Components Analysis (PCA) (Abdi & Williams, 2010) to reduce the dimension of all the raw signals and the sensory representations collected in a trajectory, we visualize them for different time steps in Figure 6.

Original: We find that the raw state signals appear to be only roughly periodical due to the noise in the states.
Revised: We find that the raw state signals appear to be only roughly periodical due to the noise in the states.

Original: However, the LSEs show stronger periodicity than the raw signals, indicating that the sensory representations can reduce the influence of the noise in raw signals, thus helping the agent better understand and react to the environment.
Revised: However, the LSEs show stronger periodicity than the raw signals, indicating that the sensory representations can reduce the influence of the noise in raw signals, thus helping the agent better understand and react to the environment.

Original: The action representations.
Revised: The action representations.

Original: To better understand the intriguing fact that the agent can adapt to new tasks using the emergent action representations when only sparse training tasks are provided, we plot the LTE space in Figure 7.
Revised: To better understand the intriguing fact that the agent can adapt to new tasks using the emergent action representations when only sparse training tasks are provided, we plot the LTE space in Figure 7.

Original: We find that the LTEs on the hypersphere automatically form a continuous space, constructing the basis of the composition of the representations in it.
Revised: We find that the LTEs on the hypersphere automatically form a continuous space, constructing the basis of the composition of the representations in it.

Original: This also explains why the interpolation and composition of the LTEs result in meaningful new behaviors: when interpolated and normalized, the new LTE would still lie on this hypersphere, leading to new and meaningful action sequences.
Revised: This also explains why the interpolation and composition of the LTEs result in meaningful new behaviors: when interpolated and normalized, the new LTE would still lie on this hypersphere, leading to new and meaningful action sequences.

Original: 4. C OMPARISON WITH ONE - HOT REPRESENTATION
Revised: 4. C OMPARISON WITH ONE - HOT REPRESENTATION

Original: Readers might be curious about whether the learned, manipulable LTEs better than a naive one-hot embedding.
Revised: Readers might be curious about whether the learned, manipulable LTEs better than a naive one-hot embedding.

Original: In this section, we compare EAR with a different policy network that takes in an onehot embedding rather then the LTE.
Revised: In this section, we compare EAR with a different policy network that takes in an onehot embedding rather then the LTE.

Original: Specifically, in multi-task training, we concatenate the raw state vector with the one-hot embedding as the input of the policy network to form a multi-task RL architecture.
Revised: Specifically, in multi-task training, we concatenate the raw state vector with the one-hot embedding as the input of the policy network to form a multi-task RL architecture.

Original: We find that the one-hot embedded policy performs unsatisfactorily in all environments and fails completely in Ant-Dir environment.
Revised: We find that the one-hot embedded policy performs unsatisfactorily in all environments and fails completely in Ant-Dir environment.

Original: We attribute this to the fact that, without the sensory-action representation, the agent fails to understand the underlying relation among multiple tasks, especially in those environments with high-dimensional observations and a large training task set.
Revised: We attribute this to the fact that, without the sensory-action representation, the agent fails to understand the underlying relation among multiple tasks, especially in those environments with high-dimensional observations and a large training task set.

Original: We also compare both methods in the task adaptation setting.
Revised: We also compare both methods in the task adaptation setting.

Original: When optimizing the one-hot embedding to adapt to new tasks, the agent can barely reach a reasonable performance in any of the new tasks.
Revised: When optimizing the one-hot embedding to adapt to new tasks, the agent can barely reach a reasonable performance in any of the new tasks.

Original: These facts echo with the idea that the simple emergent action representations are meaningful and effective in task learning and generalization.
Revised: These facts echo with the idea that the simple emergent action representations are meaningful and effective in task learning and generalization.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Representation learning in reinforcement learning.
Revised: Representation learning in reinforcement learning.

Original: Representation learning has been widely applied in RL to generate representations of sensory information (Laskin et al., 2020;
Revised: Representation learning has been widely applied in RL to generate representations of sensory information (Laskin et al., 2020;

Original: Chandak et al., 2019; Yarats et al., 2021), policies (Edwards et al., 2019), dynamics (Watter et al., 2015).
Revised: Chandak et al., 2019; Yarats et al., 2021), policies (Edwards et al., 2019), dynamics (Watter et al., 2015).

Original: In recent years, action representation has attracted more attention.
Revised: In recent years, action representation has attracted more attention.

Original: In previous works (Dulac-Arnold et al., 2015), the action embeddings have been proposed to compress discrete actions to a continuous space based on prior knowledge.
Revised: In previous works (Dulac-Arnold et al., 2015), the action embeddings have been proposed to compress discrete actions to a continuous space based on prior knowledge.

Original: Recently, many researchers focus on mapping between the original continuous action space and a continuous manifold to facilitate and accelerate policy learning (Allshire et al., 2021) or simplify teleoperation (Losey et al., 2020).
Revised: Recently, many researchers focus on mapping between the original continuous action space and a continuous manifold to facilitate and accelerate policy learning (Allshire et al., 2021) or simplify teleoperation (Losey et al., 2020).

Original: Moreover, such action representations are extended to high-dimensional real-world robotics control problems (Jeon et al., 2020).
Revised: Moreover, such action representations are extended to high-dimensional real-world robotics control problems (Jeon et al., 2020).

Original: In this area, a closely related work is Chandak et al.
Revised: In this area, a closely related work is Chandak et al.

Original: (2019), which enables a policy to output a latent action representation with reinforcement learning and then uses supervised learning to construct a mapping from the representation space to the discrete action space.
Revised: (2019), which enables a policy to output a latent action representation with reinforcement learning and then uses supervised learning to construct a mapping from the representation space to the discrete action space.

Original: However, in our work, we discover emergent self-organized sensory-action representations from multi-task policy training without additional supervised learning, which can be further leveraged to generalize to new tasks.
Revised: However, in our work, we discover emergent self-organized sensory-action representations from multi-task policy training without additional supervised learning, which can be further leveraged to generalize to new tasks.

Original: Multi-task reinforcement learning.
Revised: Multi-task reinforcement learning.

Original: Multi-task RL trains an agent to learn multiple tasks simultaneously with a single policy.
Revised: Multi-task RL trains an agent to learn multiple tasks simultaneously with a single policy.

Original: A direct idea is to share parameters and learn a joint policy with multiple objectives (Wilson et al., 2007; Pinto & Gupta, 2017), but different tasks may conflict with each other.
Revised: A direct idea is to share parameters and learn a joint policy with multiple objectives (Wilson et al., 2007; Pinto & Gupta, 2017), but different tasks may conflict with each other.

Original: Some works tackle this problem by reducing gradient conflicts (Yu et al., 2020a), designing loss weighting (Hessel et al., 2019; Kendall et al., 2018), or leveraging regularization (Duong et al., 2015).
Revised: Some works tackle this problem by reducing gradient conflicts (Yu et al., 2020a), designing loss weighting (Hessel et al., 2019; Kendall et al., 2018), or leveraging regularization (Duong et al., 2015).

Original: Modular network has also been proposed to construct the policy network with combination of sub-networks (Heess et al., 2016; Yang et al., 2020).
Revised: Modular network has also been proposed to construct the policy network with combination of sub-networks (Heess et al., 2016; Yang et al., 2020).

Original: In our work, multiple tasks share the state encoder and action decoder; the action decoder is conditioned on LTE to reduce task conflicts.
Revised: In our work, multiple tasks share the state encoder and action decoder; the action decoder is conditioned on LTE to reduce task conflicts.

Original: Emergent representations are learned for continuous actions during multi-task training.
Revised: Emergent representations are learned for continuous actions during multi-task training.

Original: Unsupervised skill discovery.
Revised: Unsupervised skill discovery.

Original: Unsupervised skill discovery employs competence-based methods to encourage exploration and help agents learn skills.
Revised: Unsupervised skill discovery employs competence-based methods to encourage exploration and help agents learn skills.

Original: These skills can be used in adapting to new tasks.
Revised: These skills can be used in adapting to new tasks.

Original: The unsupervised methods maximize the mutual information between states and skills (Eysenbach et al., 2018; Liu & Abbeel, 2021; Sharma et al., 2019; Xu et al., 2020;
Revised: The unsupervised methods maximize the mutual information between states and skills (Eysenbach et al., 2018; Liu & Abbeel, 2021; Sharma et al., 2019; Xu et al., 2020;

Original: Laskin et al., 2022), while our method learns a geometrically and semantically meaningful representation in a multi-task setting.
Revised: Laskin et al., 2022), while our method learns a geometrically and semantically meaningful representation in a multi-task setting.

Original: We believe the proposed method can be complementary to the unsupervised paradigm.
Revised: We believe the proposed method can be complementary to the unsupervised paradigm.

Original: Meta reinforcement learning.
Revised: Meta reinforcement learning.

Original: The adaptation part of this work shares a similar goal as meta reinforcement learning (meta RL)
Revised: The adaptation part of this work shares a similar goal as meta reinforcement learning (meta RL)

Original: (Finn et al., 2017; Xu et al., 2018a;
Revised: (Finn et al., 2017; Xu et al., 2018a;

Original: Rothfuss et al., 2018; Wang et al., 2016; Levine et al., 2020).
Revised: Rothfuss et al., 2018; Wang et al., 2016; Levine et al., 2020).

Original: Meta RL executes meta-learning (Schmidhuber, 1987;
Revised: Meta RL executes meta-learning (Schmidhuber, 1987;

Original: Thrun & Pratt, 2012) algorithms under the framework of reinforcement learning, which aims to train an agent to quickly adapt to new tasks.
Revised: Thrun & Pratt, 2012) algorithms under the framework of reinforcement learning, which aims to train an agent to quickly adapt to new tasks.

Original: Many meta RL methods are gradient-based, performing gradient descent on the policy (Finn et al., 2017; Xu et al., 2018a;
Revised: Many meta RL methods are gradient-based, performing gradient descent on the policy (Finn et al., 2017; Xu et al., 2018a;

Original: Rothfuss et al., 2018), the hyperparameters (Xu et al., 2018b), or loss functions (Sung et al., 2017; Houthooft et al., 2018)
Revised: Rothfuss et al., 2018), the hyperparameters (Xu et al., 2018b), or loss functions (Sung et al., 2017; Houthooft et al., 2018)

Original: based on the collected experience.
Revised: based on the collected experience.

Original: Other meta RL algorithms are context-based (Wang et al., 2016;
Revised: Other meta RL algorithms are context-based (Wang et al., 2016;

Original: Rakelly et al., 2019).
Revised: Rakelly et al., 2019).

Original: They embed context information into a latent variable and condition the meta-learning policy on such variables.
Revised: They embed context information into a latent variable and condition the meta-learning policy on such variables.

Original: Besides, recently offline meta RL algorithms (Levine et al., 2020; Mitchell et al., 2021) have also been widely explored, which enables the agent to leverage offline data to perform meta-training.
Revised: Besides, recently offline meta RL algorithms (Levine et al., 2020; Mitchell et al., 2021) have also been widely explored, which enables the agent to leverage offline data to perform meta-training.

Original: However, we exploit the emergent high-level representations to abstract low-level actions for adaptation rather than introduce complicated new techniques.
Revised: However, we exploit the emergent high-level representations to abstract low-level actions for adaptation rather than introduce complicated new techniques.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we present our finding that the task embeddings in a multi-task policy network can automatically form a space where action representations reside.
Revised: In this paper, we present our finding that the task embeddings in a multi-task policy network can automatically form a space where action representations reside.

Original: The emergent action representations abstract information of a sequence of actions and can serve as a high-level interface to instruct an agent to perform motor skills.
Revised: The emergent action representations abstract information of a sequence of actions and can serve as a high-level interface to instruct an agent to perform motor skills.

Original: Specifically, we find the action representations can be interpolated, composed, and optimized to generate novel action sequences for unseen tasks.
Revised: Specifically, we find the action representations can be interpolated, composed, and optimized to generate novel action sequences for unseen tasks.

Original: Along with this work, a promising direction is to learn the action representation via self-supervised learning.
Revised: Along with this work, a promising direction is to learn the action representation via self-supervised learning.

Original: Another intriguing direction is to learn hierarchical action representations that may capture different levels of semantics and geometric information in motor signals, thus facilitating future applications such as hierarchical planning (LeCun, 2022).
Revised: Another intriguing direction is to learn hierarchical action representations that may capture different levels of semantics and geometric information in motor signals, thus facilitating future applications such as hierarchical planning (LeCun, 2022).

Original: A E XPERIMENTAL D ETAILS A. E NVIRONMENTS
Revised: A E XPERIMENTAL D ETAILS A. E NVIRONMENTS

Original: The detailed descriptions of the reinforcement learning benchmarks in our experiments are listed as follows:
Revised: The detailed descriptions of the reinforcement learning benchmarks in our experiments are listed as follows:

Original: • HalfCheetah-Vel (Uni-modal): In this environment, we train the halfcheetah agent to run at a target velocity.
Revised: • HalfCheetah-Vel (Uni-modal): In this environment, we train the halfcheetah agent to run at a target velocity.

Original: The training task set contains 10 velocities.
Revised: The training task set contains 10 velocities.

Original: The target velocities during training range from 1 m/s to 10 m/s. For every 1 m/s, we set a training task.
Revised: The target velocities during training range from 1 m/s to 10 m/s. For every 1 m/s, we set a training task.

Original: The adaptation task set contains 3 velocities that are uniformly sampled from [1,10].
Revised: The adaptation task set contains 3 velocities that are uniformly sampled from [1,10].

Original: The agent is penalized with the lerror between its velocity and the target velocity.
Revised: The agent is penalized with the lerror between its velocity and the target velocity.

Original: • Ant-Dir (Uni-modal):
Revised: • Ant-Dir (Uni-modal):

Original: In this environment, we train the Ant agent to run in a target direction.
Revised: In this environment, we train the Ant agent to run in a target direction.

Original: The training task set contains 24 directions which are consecutive integers from 1 to 10.
Revised: The training task set contains 24 directions which are consecutive integers from 1 to 10.

Original: The adaptation task set contains 5 integer directions which are uniformly sampled from [0,360) degrees.
Revised: The adaptation task set contains 5 integer directions which are uniformly sampled from [0,360) degrees.

Original: The agent is rewarded with the velocity along the target direction and penalized with the velocity along the direction perpendicular to our target.
Revised: The agent is rewarded with the velocity along the target direction and penalized with the velocity along the direction perpendicular to our target.

Original: • Hopper/Walker-Vel
Revised: • Hopper/Walker-Vel

Original: (Uni-modal): Similar to HalfCheetah-Vel, Hopper/Walker-Vel containsvelocities in training task set and 3 in adaptation task set.
Revised: (Uni-modal): Similar to HalfCheetah-Vel, Hopper/Walker-Vel containsvelocities in training task set and 3 in adaptation task set.

Original: The target velocities when training range from 0.2 m/s to 2 m/s and every 0.2 m/s we set a training task.
Revised: The target velocities when training range from 0.2 m/s to 2 m/s and every 0.2 m/s we set a training task.

Original: The Hopper/Walker agent is penalized with the l 1 error between its velocity and the target velocity.
Revised: The Hopper/Walker agent is penalized with the l 1 error between its velocity and the target velocity.

Original: • HalfCheetah-Run-Jump
Revised: • HalfCheetah-Run-Jump

Original: Multi
Revised: Multi

Original: -modal): There are two different uni-modal task distributions in this environment, respectively HalfCheetah-BackVel and HalfCheetah-BackJump.
Revised: -modal): There are two different uni-modal task distributions in this environment, respectively HalfCheetah-BackVel and HalfCheetah-BackJump.

Original: HalfCheetahBackVel trains the halfcheetah agent to run backward at a target velocity, and the agent is penalized with the l 1 error between its velocity and the target velocity.
Revised: HalfCheetahBackVel trains the halfcheetah agent to run backward at a target velocity, and the agent is penalized with the l 1 error between its velocity and the target velocity.

Original: HalfCheetah-BackJump trains the agent to raise its hind leg to jump, and the agent is rewarded with the height of its hind leg.
Revised: HalfCheetah-BackJump trains the agent to raise its hind leg to jump, and the agent is rewarded with the height of its hind leg.

Original: HalfCheetah-Run- Jump contains 7 velocities for running backward (from 1 m/s to 7 m/s and every 1 m/s a training task is set) 
Revised: HalfCheetah-Run-Jump contains 7 velocities for running backward (from 1 m/s to 7 m/s and every 1 m/s a training task is set) and 3 tasks for jumping (3 different weights for the leg height in the total reward function) in the training task set.

Original: and

Original: 3 tasks for jumping (

Original: different

Original: weights for the leg height in the total reward function) in the training task set.

Original: In our implementation , the jumping task with smallest weight finally turns out to be a motion of “standing” instead of jumping
Revised: In our implementation, the jumping task with smallest weight finally turns out to be a motion of “standing” instead of jumping.

Original: A. B ASELINES
Revised: A. B ASELINES

Original: The detailed descriptions of the multi-task RL and meta RL baselines are listed as follows:
Revised: The detailed descriptions of the multi-task RL and meta RL baselines are listed as follows:

Original: • MAML.
Revised: • MAML.

Original: MAML (Finn et al., 2017) is a gradient-based meta-RL algorithm based on Trust Region Policy Optimization (Schulman et al., 2015).
Revised: MAML (Finn et al., 2017) is a gradient-based meta-RL algorithm based on Trust Region Policy Optimization (Schulman et al., 2015).

Original: It aims to learn easily adaptable model parameters for tasks in a specific task distribution.
Revised: It aims to learn easily adaptable model parameters for tasks in a specific task distribution.

Original: MAML conducts explicit training on model parameters so that a small number of training data and gradient steps are required to adapt to a new task.
Revised: MAML conducts explicit training on model parameters so that a small number of training data and gradient steps are required to adapt to a new task.

Original: In our implementation, we modify the range and density of task sampling, which will be discussed in the following section.
Revised: In our implementation, we modify the range and density of task sampling, which will be discussed in the following section.

Original: • PEARL.
Revised: • PEARL.

Original: PEARL (Rakelly et al., 2019) is a context-based off-policy meta-RL algorithm based on Soft Actor Critic (Haarnoja et al., 2018) and variational inference method (Kingma & Welling, 2013).
Revised: PEARL (Rakelly et al., 2019) is a context-based off-policy meta-RL algorithm based on Soft Actor Critic (Haarnoja et al., 2018) and variational inference method (Kingma & Welling, 2013).

Original: PEARL tackles the problem of limited sample efficiency in meta-RL domain.
Revised: PEARL tackles the problem of limited sample efficiency in meta-RL domain.

Original: PEARL integrates off-policy RL algorithms with a latent task variable which infers how to adapt to a new task with small amount of history contexts.
Revised: PEARL integrates off-policy RL algorithms with a latent task variable which infers how to adapt to a new task with small amount of history contexts.

Original: • Multi-head multi-task SAC (MHMT-SAC).
Revised: • Multi-head multi-task SAC (MHMT-SAC).

Original: MHMT-SAC is a multi-task RL architecture, in which each training task has an independent head input, and the body of the policy network is shared among all the tasks.
Revised: MHMT-SAC is a multi-task RL architecture, in which each training task has an independent head input, and the body of the policy network is shared among all the tasks.

Original: We compare our method with MHMT-SAC to evaluate the performance of an agent with emergent action representation in multi-task training.
Revised: We compare our method with MHMT-SAC to evaluate the performance of an agent with emergent action representation in multi-task training.

Original: • One-hot embedding SAC (OHE-SAC).
Revised: • One-hot embedding SAC (OHE-SAC).

Original: The sensory-action representation is introduced when training the agent to learn multiple tasks in the training set.
Revised: The sensory-action representation is introduced when training the agent to learn multiple tasks in the training set.

Original: In OHE-SAC, We omit the representation layer to make the policy network a common multi-task RL structure, which concatenates a one- hot embedding and the raw state vector as the input of the policy network.
Revised: In OHE-SAC, We omit the representation layer to make the policy network a common multi-task RL structure, which concatenates a one-hot embedding and the raw state vector as the input of the policy network.

Original: Through comparison between the final performance of the two methods, OHE-SAC helps identify the effect of action representations in training and adaptation.
Revised: Through comparison between the final performance of the two methods, OHE-SAC helps identify the effect of action representations in training and adaptation.

Original: A. MULTI - TASK TRAINING A.3. N ETWORK A RCHITECTURE
Revised: M A.3. N ETWORK A RCHITECTURE

Original: [Figure]
Revised: [Figure]

Original: In this section, we discuss the choice of network architecture.
Revised: In this section, we discuss the choice of network architecture.

Original: In our method, we use MLP as the network structure of the encoders and decoders.
Revised: In our method, we use MLP as the network structure of the encoders and decoders.

Original: Here we replace the first layer of MLP with an RNN layer to get the RNN encoder and decoder.
Revised: Here we replace the first layer of MLP with an RNN layer to get the RNN encoder and decoder.

Original: Our
Revised: Our

Original: MLP-based network is compared with the
Revised: MLP-based network is compared with the

Original: RNN-based network.
Revised: RNN-based network.

Original: A comparison in the
Revised: A comparison in the

Original: HalfCheetah-Vel environment of the two networks is shown in Figure 8.
Revised: HalfCheetah-Vel environment of the two networks is shown in Figure 8.

Original: We find that if we simply make such changes with a certain amount of tuning, our simple
Revised: We find that if we simply make such changes with a certain amount of tuning, our simple

Original: MLP-based network achieves much higher performance than an RNN-based one.
Revised: MLP-based network achieves much higher performance than an RNN-based one.

Original: A.3. H YPERPARAMETERS
Revised: A.3. H YPERPARAMETERS

Original: In this section, we provide detailed settings of our methods.
Revised: In this section, we provide detailed settings of our methods.

Original: We set up the hyperparameters, as shown in Table 2, for the environments and algorithms in the Mujoco locomotion benchmarks.
Revised: We set up the hyperparameters, as shown in Table 2, for the environments and algorithms in the Mujoco locomotion benchmarks.

Original: Hyperparameters
Revised: Hyperparameters

Original: HalfCheetah-Vel
Revised: HalfCheetah-Vel

Original: [Figure]

Original: /Walker-Vel Ant-
Revised: Hopper/Walker-Vel

Original: [Figure] max episode frames[Figure]   sensory encoder 2-layer MLP 2-layer MLP 2-layer MLP task encoder 1-layer MLP 1-layer MLP 1-layer MLP action decoder 4-layer MLP 4-layer MLP 4-layer MLP
Revised: Ant-Dir max episode frames   sensory encoder 2-layer MLP 2-layer MLP 2-layer MLP task encoder 1-layer MLP 1-layer MLP 1-layer MLP action decoder 4-layer MLP 4-layer MLP 4-layer MLP

Original: Q-network 4-layer MLP 4-layer MLP 4-layer MLP
Revised: Q-network 4-layer MLP 4-layer MLP 4-layer MLP

Original: LSE shape
Revised: LSE shape

Original: LTE shape   optimizer
Revised: LTE shape   optimizer

Original: Adam Adam
Revised: Adam Adam

Original: Adam learning rate   discount 
Revised: Adam learning rate   discount 0.

Original: [Table] [Table] batch size   
Revised: batch size   replay buffer 1e 1e

Original: [Table]
Revised: [Table]

Original: A.3. T ASK SAMPLING DENSITY
Revised: A.3. T ASK SAMPLING DENSITY

Original: In the experiments [Table] we fix the range of task sampling to be for different algorithms in the same environment.
Revised: In the experiments, we fix the range of task sampling to be for different algorithms in the same environment.

Original: We find that in HalfCheetah-Vel environment [Table] multi-task RL methods get different final rewards for different tasks (in Figure 9).
Revised: We find that in HalfCheetah-Vel environment, multi-task RL methods get different final rewards for different tasks (in Figure 9).

Original: The reason is that when the agent is trained well enough, the penalty of velocity in the reward will become slight, thus making the control cost of the agent dominant in the reward function.
Revised: The reason is that when the agent is trained well enough, the penalty of velocity in the reward will become slight, thus making the control cost of the agent dominant in the reward function.

Original: Therefore, it is natural that the faster the agent runs, the larger cost it should pay, leading to a relatively low reward.
Revised: Therefore, it is natural that the faster the agent runs, the larger cost it should pay, leading to a relatively low reward.

Original: Thus to make our comparison valid and persuasive, the range of task sampling should be fixed.
Revised: Thus to make our comparison valid and persuasive, the range of task sampling should be fixed.

Original: The detailed settings of the implementation in our paper are demonstrated in Table 3.
Revised: The detailed settings of the implementation in our paper are demonstrated in Table 3.

Original: Similarly, the adaptation tasks are also shared among all the algorithms in the same environment in Section 4.
Revised: Similarly, the adaptation tasks are also shared among all the algorithms in the same environment in Section 4.4.

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: Method
Revised: Method

Original: HalfCheetah-Vel
Revised: HalfCheetah-Vel

Original: Hopper/Walker-Vel
Revised: Hopper/Walker-Vel

Original: Ant-Dir
Revised: Ant-Dir

Original: EAR-SAC (Ours) 10 tasks in [0,10] 10 tasks in [0,2] 24 tasks in [0,360)
Revised: EAR-SAC (Ours) 10 tasks in [0,10] 10 tasks in [0,2] 24 tasks in [0,360)

Original: MAML 100 tasks in [0,10] 40 tasks in [0,2] 40 tasks in [0,360)
Revised: MAML 100 tasks in [0,10] 40 tasks in [0,2] 40 tasks in [0,360)

Original: PEARL 100 tasks in [0,10] 20 tasks in [0,2] 100 tasks in [0,360)
Revised: PEARL 100 tasks in [0,10] 20 tasks in [0,2] 100 tasks in [0,360)

Original: MTMH-SAC 10 tasks in [0,10] 10 tasks in [0,2] 24 tasks in [0,360)
Revised: MTMH-SAC 10 tasks in [0,10] 10 tasks in [0,2] 24 tasks in [0,360)

Original: OHE-SAC 10 tasks in [0,10] 10 tasks in [0,2] 24 tasks in [0,360)
Revised: OHE-SAC 10 tasks in [0,10] 10 tasks in [0,2] 24 tasks in [0,360)

Original: A. I NTRA - ACTION INTERPOLATION
Revised: A. I NTRA - ACTION INTERPOLATION

Original: Additional details for the “Intra-action interpolation”.
Revised: Additional details for the “Intra-action interpolation”.

Original: We have performed task interpolation between adjacent tasks in HalfCheetah-Vel in Section 4.3, where we can interpolate the LTEs of running at 1m/s and 2m/s to control the agent to run at 1.2, 1.5, 1.7m/s.
Revised: We have performed task interpolation between adjacent tasks in HalfCheetah-Vel in Section 4.3, where we can interpolate the LTEs of running at 1m/s and 2m/s to control the agent to run at 1.2, 1.5, 1.7m/s.

Original: Some visualized results are provided in the text, and here we demonstrate a quantitative analysis based on multiple seeds for the statistical reliability of our claims.
Revised: Some visualized results are provided in the text, and here we demonstrate a quantitative analysis based on multiple seeds for the statistical reliability of our claims.

Original: During the experiments, we find that the agent may learn to move forward by hand-standing instead of running when faced with the 1m/s task in some seeds.
Revised: During the experiments, we find that the agent may learn to move forward by hand-standing instead of running when faced with the 1m/s task in some seeds.

Original: It makes the postures of 1m/s and 2m/s rather different and hard to achieve interpolation.
Revised: It makes the postures of 1m/s and 2m/s rather different and hard to achieve interpolation.

Original: Due to such a phenomenon, we change the pre-trained tasks to 2m/s and 3m/s, and the interpolated tasks to 2.2m/s, 2.5m/s and 2.7m/s to fairly assess the stability of task interpolation on multiple seeds.
Revised: Due to such a phenomenon, we change the pre-trained tasks to 2m/s and 3m/s, and the interpolated tasks to 2.2m/s, 2.5m/s and 2.7m/s to fairly assess the stability of task interpolation on multiple seeds.

Original: The coefficients β of the task interpolation on 3 seeds are demonstrated in Table 4.
Revised: The coefficients β of the task interpolation on 3 seeds are demonstrated in Table 4.

Original: We can find that as the coefficient of interpolation decreases, the cheetah tends to run faster, which indicates that the
Revised: We can find that as the coefficient of interpolation decreases, the cheetah tends to run faster, which indicates that the

Original: LTEs between the embeddings of running at 2m/s and 3m/s show continuity in the LTE Space.
Revised: LTEs between the embeddings of running at 2m/s and 3m/s show continuity in the LTE Space.

Original: Mid-task interpolation.
Revised: Mid-task interpolation.

Original: Meanwhile, to fully evaluate the performance of task interpolation, we conduct another experiment on all environments.
Revised: Meanwhile, to fully evaluate the performance of task interpolation, we conduct another experiment on all environments.

Original: We aim to adjust the interpolation coefficients to achieve “mid-task interpolation” between pairs of tasks, where a “mid-task” has the average modality of the two base tasks.
Revised: We aim to adjust the interpolation coefficients to achieve “mid-task interpolation” between pairs of tasks, where a “mid-task” has the average modality of the two base tasks.

Original: For instance, in the first seed of the experiment aforementioned, we find that when β = 0 .
Revised: For instance, in the first seed of the experiment aforementioned, we find that when β = 0 .

Original: 42 , the agent can run forward at 2.5m/s, which is a mid-task of running at 2m/s and 3m/s.
Revised: 42 , the agent can run forward at 2.5m/s, which is a mid-task of running at 2m/s and 3m/s.

Original: The results of mid-task interpolation in 3 seeds are shown in Table 11,12,13,14.
Revised: The results of mid-task interpolation in 3 seeds are shown in Table 11,12,13,14.

Original: 1:1 task interpolation.
Revised: 1:1 task interpolation.

Original: In the previous parts, we first determine the desired interpolated tasks and find a suitable coefficient β for the task.
Revised: In the previous parts, we first determine the desired interpolated tasks and find a suitable coefficient β for the task.

Original: In this part, from another angle, we fix the interpolation coefficient and observe the corresponding tasks.
Revised: In this part, from another angle, we fix the interpolation coefficient and observe the corresponding tasks.

Original: We perform 1:1 task interpolation in all environments, which means we fix the coefficients β to always be 0.5.
Revised: We perform 1:1 task interpolation in all environments, which means we fix the coefficients β to always be 0.5.

Original: The results are shown in Table 15,16,17,18.
Revised: The results are shown in Table 15,16,17,18.

Original: We summarize the results of the interpolation aforementioned in Table 5.
Revised: We summarize the results of the interpolation aforementioned in Table 5.

Original: Task extrapolation trials.
Revised: Task extrapolation trials.

Original: Besides interpolation, we also perform extrapolation between tasks, which means we make β > 1 or β < 0 , in HalfCheetah-Vel and Ant-Dir.
Revised: Besides interpolation, we also perform extrapolation between tasks, which means we make β > 1 or β < 0 , in HalfCheetah-Vel and Ant-Dir.

Original: We demonstrate some of the results in Table 6.
Revised: We demonstrate some of the results in Table 6.

Original: We find that, in HalfCheetah-Vel, the extrapolated behavior can also achieve the target speed as long as the speed is in the distribution of the training tasks.
Revised: We find that, in HalfCheetah-Vel, the extrapolated behavior can also achieve the target speed as long as the speed is in the distribution of the training tasks.

Original: The exception happens when the target speed is out of the distribution(e.g.
Revised: The exception happens when the target speed is out of the distribution(e.g.

Original: extrapolate Vel-1.0 and Vel-2.0 with β > 1 or
Revised: extrapolate Vel-1.0 and Vel-2.0 with β > 1 or

Original: Vel-9.0 and Vel-10.0 with β < 0 ) where the agent can only perform extrapolation within a relatively small range of β .
Revised: Vel-9.0 and Vel-10.0 with β < 0 ) where the agent can only perform extrapolation within a relatively small range of β .

Original: In the higher-dimensional environment Ant-Dir, extrapolation still works in most cases but has a higher failure rate.
Revised: In the higher-dimensional environment Ant-Dir, extrapolation still works in most cases but has a higher failure rate.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. I NTER - ACTION COMPOSITION
Revised: A. I NTER - ACTION COMPOSITION

Original: We have performed task composition in the HalfCheetah-Run-Jump environment.
Revised: We have performed task composition in the HalfCheetah-Run-Jump environment.

Original: Two stop motion animations are shown in Section 4.3 and we provide more quantitative results here.
Revised: Two stop motion animations are shown in Section 4.3 and we provide more quantitative results here.

Original: The coefficients β of the task composition in multiple seeds are demonstrated in Table 10.
Revised: The coefficients β of the task composition in multiple seeds are demonstrated in Table 10.

Original: Composing walking and standing succeeds in all three seeds while composing running and jumping only succeeds in two seeds.
Revised: Composing walking and standing succeeds in all three seeds while composing running and jumping only succeeds in two seeds.

Original: In the seed where the run-jumping task fails, we discover a new motion called move-jumping, in which the cheetah agent jumps backward with its hind leg raised.
Revised: In the seed where the run-jumping task fails, we discover a new motion called move-jumping, in which the cheetah agent jumps backward with its hind leg raised.

Original: A. A BLATION STUDY
Revised: A. A BLATION STUDY

Original: In this section, we ablate the two constraints mentioned in Section 3.1 of our method to better understand what makes it work in the proposed structure.
Revised: In this section, we ablate the two constraints mentioned in Section 3.1 of our method to better understand what makes it work in the proposed structure.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Random Noise Injection.
Revised: Random Noise Injection.

Original: We inject random noise into the LTEs during the training procedure to enhance the smoothness of the space.
Revised: We inject random noise into the LTEs during the training procedure to enhance the smoothness of the space.

Original: To assess the effect of this injected noise, we compare it with the case without the injected noise in the high-dimensional environment Ant-Dir.
Revised: To assess the effect of this injected noise, we compare it with the case without the injected noise in the high-dimensional environment Ant-Dir.

Original: The training curves are shown in Figure 10.
Revised: The training curves are shown in Figure 10.

Original: We observe that, when injected with random noise, our method achieves comparable training performance to the one without noise.
Revised: We observe that, when injected with random noise, our method achieves comparable training performance to the one without noise.

Original: Then we evaluate the policy we learn.
Revised: Then we evaluate the policy we learn.

Original: For each task in each seed, we evaluate the trained policy for multiple trajectories.
Revised: For each task in each seed, we evaluate the trained policy for multiple trajectories.

Original: We find that without the injected noise in training, the policy will be sensitive to perturbation in the system during evaluation.
Revised: We find that without the injected noise in training, the policy will be sensitive to perturbation in the system during evaluation.

Original: An example of such a phenomenon is shown in Table 7.
Revised: An example of such a phenomenon is shown in Table 7.

Original: In the task of “direction-240” in Ant-Dir, we evaluate the policy fortrajectories and find that the second and third trials terminate before the maximal steps (200 steps), which means they fail halfway.
Revised: In the task of “direction-240” in Ant-Dir, we evaluate the policy fortrajectories and find that the second and third trials terminate before the maximal steps (200 steps), which means they fail halfway.

Original: We count the number of such failures respectively and summarize them in Table 8.
Revised: We count the number of such failures respectively and summarize them in Table 8.

Original: The policy trained without injected noise fails to perform over 20% of tasks stably, while the one with noise avoids failure.
Revised: The policy trained without injected noise fails to perform over 20% of tasks stably, while the one with noise avoids failure.

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: Unit sphere regularization.
Revised: Unit sphere regularization.

Original: We also apply regularization on LTEs by normalizing them on the surface of a unit sphere.
Revised: We also apply regularization on LTEs by normalizing them on the surface of a unit sphere.

Original: We ablate this regularization by removing the normalization functions for LSEs and LTEs and directly concatenating them together as the input of the action decoder.
Revised: We ablate this regularization by removing the normalization functions for LSEs and LTEs and directly concatenating them together as the input of the action decoder.

Original: The training curves are demonstrated in Figure 11.
Revised: The training curves are demonstrated in Figure 11.

Original: We find that our method achieves higher sample efficiency and rewards than the one without normalization.
Revised: We find that our method achieves higher sample efficiency and rewards than the one without normalization.

Original: Then we compare their performance in downstream tasks.
Revised: Then we compare their performance in downstream tasks.

Original: We repeat the interpolation experiments with the new policy without normalization and calculate the success rates in Table 9.
Revised: We repeat the interpolation experiments with the new policy without normalization and calculate the success rates in Table 9.

Original: With the normalization, the success rates of interpolation experiments increase, which indicates that normalization improves the quality of the representation space and thus better supports interpolation.
Revised: With the normalization, the success rates of interpolation experiments increase, which indicates that normalization improves the quality of the representation space and thus better supports interpolation.

Original: Furthermore, we also try to perform task adaptation with the new policy.
Revised: Furthermore, we also try to perform task adaptation with the new policy.

Original: An example is demonstrated in Figure 12.
Revised: An example is demonstrated in Figure 12.

Original: Results show that the new policy can still adapt to some tasks while requiring more training steps.
Revised: Results show that the new policy can still adapt to some tasks while requiring more training steps.

Original: [Table]
Revised: [Table]

Revised: With normalization 70/ 66/

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A. A PPLICATION IN MANIPULATION
Revised: A. A PPLICATION IN MANIPULATION

Original: In previous sections [Figure] we mainly tackle the classic and widely studied locomotion problems in reinforcement learning.
Revised: In previous sections, we mainly tackle the classic and widely studied locomotion problems in reinforcement learning.

Original: In this section, we apply our method to the domain of manipulation to examine the effectiveness of the proposed method in other domains.
Revised: In this section, we apply our method to the domain of manipulation to examine the effectiveness of the proposed method in other domains.

Original: We propose another environment
Revised: We propose another environment

Original: Metaworld-Reach based on the metaworld (Yu et al., 2020b) [Figure] in which a robot arm is trained to reach different points.
Revised: Metaworld-Reach based on the metaworld (Yu et al., 2020b), in which a robot arm is trained to reach different points.

Original: We train the agent with multiple reach goals under the proposed multi-task setting and perform fast adaptation with the learned policy.
Revised: We train the agent with multiple reach goals under the proposed multi-task setting and perform fast adaptation with the learned policy.

Original: The training and adaptation curves are demonstrated respectively in Figure 13,14.
Revised: The training and adaptation curves are demonstrated respectively in Figure 13,14.

Original: We find that our method can still find suitable latent task embeddings in the action representation space for adaptation tasks in less than 5 steps. 
Revised: We find that our method can still find suitable latent task embeddings in the action representation space for adaptation tasks in less than 5 steps. [Table] [Table] [Table]

Revised: Target

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Target

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Evaluation

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Target

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Evaluation

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: Target

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: Evaluation

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: Target

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: Evaluation

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]