Original: Human-AI Shared Control via Policy Dissection
Revised: Human-AI Shared Control via Policy Dissection

Original: Abstract
Revised: Abstract

Original: Human-AI shared control allows human to interact and collaborate with autonomous agents to accomplish control tasks in complex environments.
Revised: Human-AI shared control allows human to interact and collaborate with autonomous agents to accomplish control tasks in complex environments.

Original: Previous Reinforcement Learning (RL) methods attempted goal-conditioned designs to achieve human-controllable policies at the cost of redesigning the reward function and training paradigm.
Revised: Previous Reinforcement Learning (RL) methods attempted goal-conditioned designs to achieve human-controllable policies at the cost of redesigning the reward function and training paradigm.

Original: Inspired by the neuroscience approach to investigate the motor cortex in primates, we develop a simple yet effective frequency-based approach called Policy Dissection to align the intermediate representation of the learned neural controller with the kinematic attributes of the agent behavior.
Revised: Inspired by the neuroscience approach to investigate the motor cortex in primates, we develop a simple yet effective frequency-based approach called Policy Dissection to align the intermediate representation of the learned neural controller with the kinematic attributes of the agent behavior.

Original: Without modifying the neural controller or retraining the model, the proposed approach can convert a given RL-trained policy into a human-controllable policy.
Revised: Without modifying the neural controller or retraining the model, the proposed approach can convert a given RL-trained policy into a human-controllable policy.

Original: We evaluate the proposed approach on many RL tasks such as autonomous driving and locomotion.
Revised: We evaluate the proposed approach on many RL tasks such as autonomous driving and locomotion.

Original: The experiments show that human-AI shared control system achieved by Policy Dissection in driving task can substantially improve the performance and safety in unseen traffic scenes.
Revised: The experiments show that human-AI shared control system achieved by Policy Dissection in driving task can substantially improve the performance and safety in unseen traffic scenes.

Original: With human in the inference loop, the locomotion robots also exhibit versatile controllable motion skills even though they are only trained to move forward.
Revised: With human in the inference loop, the locomotion robots also exhibit versatile controllable motion skills even though they are only trained to move forward.

Original: Our results suggest the promising direction of implementing human-AI shared autonomy through interpreting the learned representation of the autonomous agents.
Revised: Our results suggest the promising direction of implementing human-AI shared autonomy through interpreting the learned representation of the autonomous agents.

Original: Code and demo videos are available at https://metadriverse.github.io/policydissect .
Revised: Code and demo videos are available at https://metadriverse.github.io/policydissect .

Original:  Introduction
Revised:  Introduction

Original: In recent years, autonomous agents trained from deep Reinforcement Learning (RL) have achieved huge success in a range of applications from robot control [69, 56, 37, 41, 73], autonomous driving [25, 7, 29] to the power system in smart building [44].
Revised: In recent years, autonomous agents trained from deep Reinforcement Learning (RL) have achieved huge success in a range of applications from robot control [68, 56, 37, 41, 72], autonomous driving [25, 7, 29], to the power system in smart building [44].

Original: Despite the capability of discovering feasible policy under unknown system dynamics in a model-free setting [67], the learning-based agents lack sufficient generalizability [14] in unseen environments, which hinders their real-world deployments.
Revised: Despite the capability of discovering feasible policy under unknown system dynamics in a model-free setting [66], the learning-based agents lack sufficient generalizability [14] in unseen environments, which hinders their real-world deployments.

Original: Furthermore, it remains difficult to understand the internal mechanism and the decision-making of the neural network [30, 79], especially when abnormal behaviors happen [10, 78, 49].
Revised: Furthermore, it remains difficult to understand the internal mechanism and the decision-making of the neural network [30, 78], especially when abnormal behaviors happen [10, 77, 49].

Original: Thus the absence of generalizability, transparency and controllability limits the application of the end-to-end neural controllers in safety-critical systems.
Revised: Thus the absence of generalizability, transparency and controllability limits the application of the end-to-end neural controllers in safety-critical systems.

Original: One potential direction to make the neural controller safer and more trustworthy is incorporating human into the inference loop, which we refer to as human-AI shared control .
Revised: One potential direction to make the neural controller safer and more trustworthy is incorporating human into the inference loop, which we refer to as human-AI shared control .

Original: With human involvement, the shared autonomy systems can achieve substantial improvement over the performance and safety [54, 15, 55], even on new tasks and unseen environments.
Revised: With human involvement, the shared autonomy systems can achieve substantial improvement over the performance and safety [54, 15, 55], even on new tasks and unseen environments.

Original: Human-AI shared control has been implemented in various forms, such as enforcing intervention [28, 51] or providing high-level commands [15] to the AI models.
Revised: Human-AI shared control has been implemented in various forms, such as enforcing intervention [28, 51] or providing high-level commands [15] to the AI models.

Original: Previous works mainly explore goal-conditioned reinforcement learning (GCRL) as a form of human-AI shared control, where a high-level goal is fed as an input to the policy during training [60, 2, 15, 25, 57].
Revised: Previous works mainly explore goal-conditioned reinforcement learning (GCRL) as a form of human-AI shared control, where a high-level goal is fed as an input to the policy during training [60, 2, 15, 25, 57].

Original: Thus the behavior of the learned agent can be controlled by varying the input goal [47, 38, 80, 37].
Revised: Thus the behavior of the learned agent can be controlled by varying the input goal [47, 38, 79, 37].

Original: However, training goal-conditioned policy requires additional modifications to the observation design, the reward function and the training
Revised: However, training goal-conditioned policy requires additional modifications to the observation design, the reward function and the training

Original: [Figure]
Revised: [Figure]

Original: Pybullet-A
Revised: Pybullet-A

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: process.
Revised: process.

Original: In the case of bipedal robot for example, it is difficult to design reward functions for each behavior like crouching, backflipping, forward jumping.
Revised: In the case of bipedal robot for example, it is difficult to design reward functions for each behavior like crouching, backflipping, forward jumping.

Original: Also, initial state distribution and goal selection criteria should be meticulously designed for a successful GCRL training [50, 57].
Revised: Also, initial state distribution and goal selection criteria should be meticulously designed for a successful GCRL training [50, 57].

Original: On the other hand, the human control only happens at the goal level, still lacking in the interpretability of low-level neural policy.
Revised: On the other hand, the human control only happens at the goal level, still lacking in the interpretability of low-level neural policy.

Original: In this work, we develop a minimalist approach called Policy Dissection , to enable human-AI shared control on autonomous agents trained on a wide range of tasks.
Revised: In this work, we develop a minimalist approach called Policy Dissection , to enable human-AI shared control on autonomous agents trained on a wide range of tasks.

Original: Policy Dissection does not impose assumptions on the agent’s training scheme like specially-designed reward function, and it requires neither retraining the controller’s network nor modifying the environment.
Revised: Policy Dissection does not impose assumptions on the agent’s training scheme like specially-designed reward function, and it requires neither retraining the controller’s network nor modifying the environment.

Original: Policy Dissection achieves human-AI shared control by firstly dissecting the internal representations of the learned policy and aligning them with specific kinematic attributes, and then modulating the activation of the internal representations to elicit the desired motor skills.
Revised: Policy Dissection achieves human-AI shared control by firstly dissecting the internal representations of the learned policy and aligning them with specific kinematic attributes, and then modulating the activation of the internal representations to elicit the desired motor skills.

Original: Policy Dissection is inspired by the neuroscience studies on motor cortex of primates, where exerting electrical stimulation on different areas of motor cortex can elicit meaningful body movements [21].
Revised: Policy Dissection is inspired by the neuroscience studies on motor cortex of primates, where exerting electrical stimulation on different areas of motor cortex can elicit meaningful body movements [21].

Original: Thus, a stimulation-evoked map can be built to reveal the relationship between the evoked body movements and the motor neurons located in different areas of motor cortex [21, 20, 34].
Revised: Thus, a stimulation-evoked map can be built to reveal the relationship between the evoked body movements and the motor neurons located in different areas of motor cortex [21, 20, 34].

Original: In order to replicate such a stimulation-evoked map inside a controller powered by artificial neural network, the proposed Policy Dissection conducts a frequency analysis and matching strategy for associating the kinematic attribute and the unit activation.
Revised: In order to replicate such a stimulation-evoked map inside a controller powered by artificial neural network, the proposed Policy Dissection conducts a frequency analysis and matching strategy for associating the kinematic attribute and the unit activation.

Original: The procedure of building a stimulation-evoked map by
Revised: The procedure of building a stimulation-evoked map by

Original: Policy Dissection is illustrated in Fig.
Revised: Policy Dissection is illustrated in Fig.

Original: Concretely, we first roll out the given well-trained policy for many episodes and collect a set of time series of the activities of all units, and the variation in kinematics and dynamics.
Revised: Concretely, we first roll out the given well-trained policy for many episodes and collect a set of time series of the activities of all units, and the variation in kinematics and dynamics.

Original: Afterwards, for each kinematic attribute we identify one unit whose activity align best with the variation of this kinematic attribute following the principle that there is the lowest frequency discrepancy between both the time series.
Revised: Afterwards, for each kinematic attribute we identify one unit whose activity align best with the variation of this kinematic attribute following the principle that there is the lowest frequency discrepancy between both the time series.

Original: After finishing the alignment for
Revised: After finishing the alignment for

Original: [Figure]
Revised: [Figure]

Original: all kinematic attributes, the identified units are then called motor primitives 1 whose activation can change corresponding kinematic attributes.
Revised: all kinematic attributes, the identified units are then called motor primitives 1 whose activation can change corresponding kinematic attributes.

Original: Since a behavior can be described by the change of one or more kinematic attributes, the stimulation-evoked map can be built for recording the relationship of motor-primitives ’ activation and evoked behaviors.
Revised: Since a behavior can be described by the change of one or more kinematic attributes, the stimulation-evoked map can be built for recording the relationship of motor-primitives ’ activation and evoked behaviors.

Original: For example, the moving up behavior of the Gym-Ant shown in Fig.
Revised: For example, the moving up behavior of the Gym-Ant shown in Fig.

Original: 2 can be described by turning upward (increasing yaw) and increasing upward speed ( v y ), and thus we can activate corresponding two motor primitives to elicit the desired movement.
Revised: 2 can be described by turning upward (increasing yaw) and increasing upward speed ( v y ), and thus we can activate corresponding two motor primitives to elicit the desired movement.

Original: In conclusion, Policy Dissection constructs an interpretable control interface on top of the trained agent for human to interact with and evoke certain behaviors of the agent, thus achieving human-AI shared control.
Revised: In conclusion, Policy Dissection constructs an interpretable control interface on top of the trained agent for human to interact with and evoke certain behaviors of the agent, thus achieving human-AI shared control.

Original: We evaluate the proposed method on several RL agents ranging from locomotion robots to autonomous driving agents in simulation environments.
Revised: We evaluate the proposed method on several RL agents ranging from locomotion robots to autonomous driving agents in simulation environments.

Original: Experimental results suggest that meaningful motor primitives emerge in the internal representation.
Revised: Experimental results suggest that meaningful motor primitives emerge in the internal representation.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, novel behaviors can be deliberately evoked by activating units related to certain kinematic attributes, even though they are not the necessary behaviors to solve the primal task.
Revised: 1, novel behaviors can be deliberately evoked by activating units related to certain kinematic attributes, even though they are not the necessary behaviors to solve the primal task.

Original: In the quantitative evaluation, we use the humanAI shared control system enabled by Policy Dissection to improve the generalization in test-time unseen environment and achieve zero-shot task transfer.
Revised: In the quantitative evaluation, we use the humanAI shared control system enabled by Policy Dissection to improve the generalization in test-time unseen environment and achieve zero-shot task transfer.

Original: In an autonomous driving task, we first train baseline agents under mild traffic conditions and evaluate the driving policy on new test environments containing unseen near-accidental scenes.
Revised: In an autonomous driving task, we first train baseline agents under mild traffic conditions and evaluate the driving policy on new test environments containing unseen near-accidental scenes.

Original: Different from the poor performance of the baseline agent in these new environments, the human-AI shared control system achieves superior performance and safety guarantee, with 95% success rate and almost zero cost.
Revised: Different from the poor performance of the baseline agent in these new environments, the human-AI shared control system achieves superior performance and safety guarantee, with 95% success rate and almost zero cost.

Original: In the quadrupedal locomotion task, a robot with only proprioceptive state input can avoid obstacles with shared control, even though its primal task is moving forward.
Revised: In the quadrupedal locomotion task, a robot with only proprioceptive state input can avoid obstacles with shared control, even though its primal task is moving forward.

Original: These results show that the proposed Policy Dissection not only helps understand the learned representations of the neural controller, but also brings the intriguing potential application of human-AI shared control.
Revised: These results show that the proposed Policy Dissection not only helps understand the learned representations of the neural controller, but also brings the intriguing potential application of human-AI shared control.

Original:  Related Work
Revised:  Related Work

Original: Human-AI Shared Control .
Revised: Human-AI Shared Control .

Original: Existing methods of human-AI shared control can be roughly divided into two categories.
Revised: Existing methods of human-AI shared control can be roughly divided into two categories.

Original: The first category is to have human in the training loop and then the welltrained policy can be executed without human in test time [81].
Revised: The first category is to have human in the training loop and then the welltrained policy can be executed without human in test time [80].

Original: By incorporating human into the training, previous works successfully improve the performance on visual control tasks such as Atari game [1, 58, 71].
Revised: By incorporating human into the training, previous works successfully improve the performance on visual control tasks such as Atari game [1, 58, 70].

Original: Robotic control tasks also benefit from human feedback [28, 43, 65, 51, 72].
Revised: Robotic control tasks also benefit from human feedback [28, 43, 64, 51, 71].

Original: The other category is to have human in both training and test time to accurately accomplish humanassistive tasks.
Revised: The other category is to have human in both training and test time to accurately accomplish humanassistive tasks.

Original: A series of works study this problem based on the Atari Game [54, 59, 8, 31].
Revised: A series of works study this problem based on the Atari Game [54, 59, 8, 31].

Original: In addition, the human-AI shared control system is built on autonomous vehicle [19], robotic arms [27] and in multi-agent setting [31].
Revised: In addition, the human-AI shared control system is built on autonomous vehicle [19], robotic arms [27] and in multi-agent setting [31].

Original: The reported results on various tasks indicate the effectiveness and efficiency of training and coordination with human-assistive AI.
Revised: The reported results on various tasks indicate the effectiveness and efficiency of training and coordination with human-assistive AI.

Original: Unlike prior works, our method
Revised: Unlike prior works, our method

Original: [Figure]
Revised: [Figure]

Original: Neural Network Interpretability .
Revised: Neural Network Interpretability .

Original: There is a growing interest in the AI community to understand what is learned inside the neural network.
Revised: There is a growing interest in the AI community to understand what is learned inside the neural network.

Original: Previous interpretability methods mostly focus on explaining the networks trained for visual tasks [82, 63, 11, 83, 40].
Revised: Previous interpretability methods mostly focus on explaining the networks trained for visual tasks [81, 62, 11, 82, 40].

Original: Various empirical studies have found that meaningful and interpretable structures emerge in the internal representation when trained for the primal tasks.
Revised: Various empirical studies have found that meaningful and interpretable structures emerge in the internal representation when trained for the primal tasks.

Original: For example, object detectors emerge in scene classifier [6], and disentangled layout and attribute controllers emerge from image generation model [75].
Revised: For example, object detectors emerge in scene classifier [6], and disentangled layout and attribute controllers emerge from image generation model [74].

Original: Varying the disentangled hidden features of the deep generative model can edit the content of the generated images.
Revised: Varying the disentangled hidden features of the deep generative model can edit the content of the generated images.

Original: On the other hand, visuomotor control tasks can also acquire visual understanding to explain the behavior chosen by agent with visual input through saliency map [66, 3, 53, 12, 26, 22].
Revised: On the other hand, visuomotor control tasks can also acquire visual understanding to explain the behavior chosen by agent with visual input through saliency map [65, 3, 53, 12, 26, 22].

Original: Neuroscience Inspired Control .
Revised: Neuroscience Inspired Control .

Original: The complex behaviors shown by recent autonomous agents prompt researchers to develop new analysis methods.
Revised: The complex behaviors shown by recent autonomous agents prompt researchers to develop new analysis methods.

Original: As a source of inspiration, neuroscience provides useful tools [48, 34], and effective procedures [24, 18, 46] for understanding animal brains which are also like black-box systems.
Revised: As a source of inspiration, neuroscience provides useful tools [48, 34], and effective procedures [24, 18, 46] for understanding animal brains which are also like black-box systems.

Original: Recent works try to develop neural network-based policies from the perspective of neuroscience [17, 4, 77, 13].
Revised: Recent works try to develop neural network-based policies from the perspective of neuroscience [17, 4, 76, 13].

Original: In a closely related work on understanding learned agent,
Revised: In a closely related work on understanding learned agent,

Original: Merel et al.
Revised: Merel et al.

Original: [46] use RSA [35] with CKA [33] similarity index to reveal that behaviors spanning multiple timescales are encoded in distinct neural layers.
Revised: [46] use RSA [35] with CKA [33] similarity index to reveal that behaviors spanning multiple timescales are encoded in distinct neural layers.

Original: Apart from providing the neuroscientific tools, many concepts from neuroscience, such as motor primitive also inspire researchers to design generalizable controllers [70, 68, 45].
Revised: Apart from providing the neuroscientific tools, many concepts from neuroscience, such as motor primitive also inspire researchers to design generalizable controllers [69, 67, 45].

Original:  Policy Dissection Method
Revised:  Policy Dissection Method

Original: Policy Dissection aims at deriving a human-controllable policy from a well-trained RL agent.
Revised: Policy Dissection aims at deriving a human-controllable policy from a well-trained RL agent.

Original: It has two unique features to enable human-AI shared control: First, it operates on a well-trained RL agent without re-training or modifying the original reward/environment and the policy architecture.
Revised: It has two unique features to enable human-AI shared control: First, it operates on a well-trained RL agent without re-training or modifying the original reward/environment and the policy architecture.

Original: We only need to roll out the agent for several episodes and log its neural activity and the change of kinematic attributes.
Revised: We only need to roll out the agent for several episodes and log its neural activity and the change of kinematic attributes.

Original: Based on the collected data, kinematic attributes are aligned with certain units which are further termed as motor primitives .
Revised: Based on the collected data, kinematic attributes are aligned with certain units which are further termed as motor primitives .

Original: Thus our method is non-intrusive to the primal task and trained model, and applicable to a wide range of environments.
Revised: Thus our method is non-intrusive to the primal task and trained model, and applicable to a wide range of environments.

Original: Second, activating certain motor primitives can elicit generalizable motor skills that even the agent is not trained to perform.
Revised: Second, activating certain motor primitives can elicit generalizable motor skills that even the agent is not trained to perform.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, the bipedal Cassie robot can be steered to tip toe, crouch and back flip even if it is only trained to move forward in the primal task.
Revised: 1, the bipedal Cassie robot can be steered to tip toe, crouch and back flip even if it is only trained to move forward in the primal task.

Original: In the following subsections, we will introduce the four essential steps of the Policy Dissection method in detail.
Revised: In the following subsections, we will introduce the four essential steps of the Policy Dissection method in detail.

Original: 3.1 Monitoring Neural Activity and Kinematics
Revised: 3.1 Monitoring Neural Activity and Kinematics

Original: Since multi-layer perceptron (MLP) is the most common network architecture used in many RL tasks, we assume the neural controller is an MLP with Z units, which has L hidden layers and all layers have the same number of I hidden units 2 .
Revised: Since multi-layer perceptron (MLP) is the most common network architecture used in many RL tasks, we assume the neural controller is an MLP with Z units, which has L hidden layers and all layers have the same number of I hidden units 2 .

Original: We denote a neuron by l, i as z l,
Revised: We denote a neuron by l, i as z l,

Original: i ∈
Revised: i ∈

Original: Z , where l = 1 , ..., L is the index of layer and i = 1 , ..., I is the neuron index in each layer, and its output at time step t is z l,it .
Revised: Z , where l = 1 , ..., L is the index of layer and i = 1 , ..., I is the neuron index in each layer, and its output at time step t is z l,it .

Original: Let J be the total number of kinematic attributes.
Revised: Let J be the total number of kinematic attributes.

Original: The agent’s kinematic attributes are represented by S =
Revised: The agent’s kinematic attributes are represented by S =

Original: { s j }, j = 1 , ..., J , and the measurement of each kinematic attribute s j at time step t is s jt .
Revised: { s j }, j = 1 , ..., J , and the measurement of each kinematic attribute s j at time step t is s jt .

Original: We define the motor primitive as the unit m j ∈ Z that can steer the agent and cause a change of the corresponding kinematic attribute.
Revised: We define the motor primitive as the unit m j ∈ Z that can steer the agent and cause a change of the corresponding kinematic attribute.

Original: We assume that each kinematic attribute s j has a corresponding motor primitive
Revised: We assume that each kinematic attribute s j has a corresponding motor primitive

Original: m j to drive the attribute to a desired state, and thus there are also J motor primitives , denoted by M = { m j } , j = 1 , ..., J .
Revised: m j to drive the attribute to a desired state, and thus there are also J motor primitives , denoted by M = { m j } , j = 1 , ..., J .

Original: Our goal is to discover the motor primitive m j by associating the units with certain kinematic attribute s j .
Revised: Our goal is to discover the motor primitive m j by associating the units with certain kinematic attribute s j .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, we take the Ant robot as an example.
Revised: 3, we take the Ant robot as an example.

Original: Policy Dissection starts by rolling out the pretrained agent for several episodes and tracking the activation of all units as well as the kinematics, such as yaw rate and velocity.
Revised: Policy Dissection starts by rolling out the pretrained agent for several episodes and tracking the activation of all units as well as the kinematics, such as yaw rate and velocity.

Original: We record the I · L time series { z l,i =
Revised: We record the I · L time series { z l,i =

Original: [ z l,i 1 , z l,i 2 , ... ] } I · L measuring the neural activities and J time series of kinematic attributes { s j =
Revised: [ z l,i 1 , z l,i 2 , ... ] } I · L measuring the neural activities and J time series of kinematic attributes { s j =

Original: [ s j 1 , s j 2 , ... ] } J for further analysis.
Revised: [ s j 1 , s j 2 , ... ] } J for further analysis.

Original: 3.2 Associating Units with Kinematic Attributes
Revised: 3.2 Associating Units with Kinematic Attributes

Original: Observing the neural activity and kinematic measurement recorded in Fig.
Revised: Observing the neural activity and kinematic measurement recorded in Fig.

Original: 3, we can find that several repeated kinematic patterns appear in different frequencies when executing the policy, which is reminiscent of animal behavioral research [21, 20, 34].
Revised: 3, we can find that several repeated kinematic patterns appear in different frequencies when executing the policy, which is reminiscent of animal behavioral research [21, 20, 34].

Original: It suggests that the motor skills has its predominant frequency component [46], controlling fast kinematics and slow kinematics.
Revised: It suggests that the motor skills has its predominant frequency component [46], controlling fast kinematics and slow kinematics.

Original: Since neural activation signals also encode distinct frequency information as shown in Fig.
Revised: Since neural activation signals also encode distinct frequency information as shown in Fig.

Original: 3, this resemblance inspires us to make the following hypothesize:
Revised: 3, this resemblance inspires us to make the following hypothesize:

Original: For one kinematic attribute, the unit with minimal frequency discrepancy is the motor primitive .
Revised: For one kinematic attribute, the unit with minimal frequency discrepancy is the motor primitive .

Original: To this end, we process time series of neural activity and kinematic attributes in frequency domain, where the difference of predominant frequency between two signals can be clearly revealed and the phase discrepancy of two time series are eliminated.
Revised: To this end, we process time series of neural activity and kinematic attributes in frequency domain, where the difference of predominant frequency between two signals can be clearly revealed and the phase discrepancy of two time series are eliminated.

Original: As a consequence, this allows us to analyze the recorded data and associate kinematic attributes with units whose activity best matches the variation of target kinematic attributes.
Revised: As a consequence, this allows us to analyze the recorded data and associate kinematic attributes with units whose activity best matches the variation of target kinematic attributes.

Original: Concretely, all time series { x } including the neural activities { z l,i } and change of kinematic attributes { s j } are transformed into the frequency domain through Discrete-time
Revised: Concretely, all time series { x } including the neural activities { z l,i } and change of kinematic attributes { s j } are transformed into the frequency domain through Discrete-time

Original: Short-time Fourier transform (STFT):
Revised: Short-time Fourier transform (STFT):

Original: [Equation]
Revised: [Equation]

Original: STFT splits the time series x = [ x 1 , x 2 , ... ] into a sequence of temporal window and d ∈ [1 , + ∞ ] denotes the time step of the center of the temporal window.
Revised: STFT splits the time series x = [ x 1 , x 2 , ... ] into a sequence of temporal window and d ∈ [1 , + ∞ ] denotes the time step of the center of the temporal window.

Original: STFT then conducts Fourier transform in each temporal window.
Revised: STFT then conducts Fourier transform in each temporal window.

Original: ω ∈ [0 , + ∞ ] denotes the frequency.
Revised: ω ∈ [0 , + ∞ ] denotes the frequency.

Original: To reduce spectral leakage and reserve the frequency information as much as possible, the window function W ( t ) used in Policy Dissection is blackman window .
Revised: To reduce spectral leakage and reserve the frequency information as much as possible, the window function W ( t ) used in Policy Dissection is blackman window .

Original: Moreover, the window length is 64 with hop length 16 .
Revised: Moreover, the window length is 64 with hop length 16 .

Original: We apply STFT instead of naive Fourier transform since it can retain temporal information by breaking the time series into several overlapping chunks of frames.
Revised: We apply STFT instead of naive Fourier transform since it can retain temporal information by breaking the time series into several overlapping chunks of frames.

Original: This feature is important since in the following frequency matching we focus on the similarity of resulting spectrograms as well as the temporal correlation of two signals.
Revised: This feature is important since in the following frequency matching we focus on the similarity of resulting spectrograms as well as the temporal correlation of two signals.

Original: We use the spectrogram to characterize a measurement:
Revised: We use the spectrogram to characterize a measurement:

Original: [Equation]
Revised: [Equation]

Original: After obtaining the spectrograms for all kinematic attributes and neural activities, frequency discrepancy can be computed for each neuron-kinematics pair ( z l,i , s j ) :
Revised: After obtaining the spectrograms for all kinematic attributes and neural activities, frequency discrepancy can be computed for each neuron-kinematics pair ( z l,i , s j ) :

Original: [Equation]
Revised: [Equation]

Original: For each kinematic attribute, frequency matching is applied to select the unit with minimal average discrepancy across N collected episodes as the motor primitive responsible for triggering the change of desired kinematic attributes:
Revised: For each kinematic attribute, frequency matching is applied to select the unit with minimal average discrepancy across N collected episodes as the motor primitive responsible for triggering the change of desired kinematic attributes:

Original: [Equation]
Revised: [Equation]

Original: 3.3 Building Stimulation-evoked Map
Revised: 3.3 Building Stimulation-evoked Map

Original: A behavior b k can be described by changing a subset of kinematic attributes S k ⊆ S , which can be achieved by activating a set of corresponding motor primitives M k ⊆ M .
Revised: A behavior b k can be described by changing a subset of kinematic attributes S k ⊆ S , which can be achieved by activating a set of corresponding motor primitives M k ⊆ M .

Original: Therefore, these movement generation building blocks, motor primitives , are associated with certain behaviors, inducing the stimulation-evoked map { ( M k , b k ) } , k
Revised: Therefore, these movement generation building blocks, motor primitives , are associated with certain behaviors, inducing the stimulation-evoked map { ( M k , b k ) } , k

Original: = 1 , ..., K , where K is the number of discovered behaviors.
Revised: = 1 , ..., K , where K is the number of discovered behaviors.

Original: Taking Cassie’s back-flip shown in Fig.
Revised: Taking Cassie’s back-flip shown in Fig.

Original: 1 as an example, this behavior can be described by increasing 1 .
Revised: 1 as an example, this behavior can be described by increasing 1 .

Original: height 2 . pitch and 3 .
Revised: height 2 . pitch and 3 .

Original: knee force.
Revised: knee force.

Original: Therefore, we activate three corresponding motor primitives selected by Eq.
Revised: Therefore, we activate three corresponding motor primitives selected by Eq.

Original: 4 with output calculated by Eq.
Revised: 4 with output calculated by Eq.

Original: 8 to make the robot back flip.
Revised: 8 to make the robot back flip.

Original: In practice, it is possible that more motor primitives are required.
Revised: In practice, it is possible that more motor primitives are required.

Original: Again, taking the back-flip as an example, the force for both knee may be different after activating corresponding units, and thus giving the robot a rolling velocity, which can be further eliminated by activating roll related motor primitive .
Revised: Again, taking the back-flip as an example, the force for both knee may be different after activating corresponding units, and thus giving the robot a rolling velocity, which can be further eliminated by activating roll related motor primitive .

Original: 3.4 Steering Agent via Stimulation-evoked Map
Revised: 3.4 Steering Agent via Stimulation-evoked Map

Original: Now the last thing for evoking a behavior b k is to determine the output value for motor primitives S k .
Revised: Now the last thing for evoking a behavior b k is to determine the output value for motor primitives S k .

Original: At time step t , activating the motor primitive m j related to kinematic attribute s j stands for applying its derivative ˙
Revised: At time step t , activating the motor primitive m j related to kinematic attribute s j stands for applying its derivative ˙

Original: s j to the agent.
Revised: s j to the agent.

Original: For example, activating the speed related units equals to applying the derivative of speed, acceleration, to the agent.
Revised: For example, activating the speed related units equals to applying the derivative of speed, acceleration, to the agent.

Original: Therefore, changing the kinematic attribute s j at time step t 0 by activating the motor primitive m j with a unit output v j can be formulated as:
Revised: Therefore, changing the kinematic attribute s j at time step t 0 by activating the motor primitive m j with a unit output v j can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: where T = t 1 − t 0 is the activation period of the motor primitive
Revised: where T = t 1 − t 0 is the activation period of the motor primitive

Original: m j
Revised: m j

Original: , constant v j is the time-invariant output of m j and f ( v ) is a mapping from motor primitive output value v j to ˙
Revised: , constant v j is the time-invariant output of m j and f ( v ) is a mapping from motor primitive output value v j to ˙

Original: s j .
Revised: s j .

Original: For steering the agent, our goal is to understand some properties of f ( v ) , especially the correlation between f ( v ) and v , so that we can amplify or reduce the s j by choosing a v or − v .
Revised: For steering the agent, our goal is to understand some properties of f ( v ) , especially the correlation between f ( v ) and v , so that we can amplify or reduce the s j by choosing a v or − v .

Original: Taking the Ant robot of Fig.
Revised: Taking the Ant robot of Fig.

Original: 1 as an example, activating the unit controlling y-axis movement with positive value can drive the Ant move left, and a negative value makes the Ant move right.
Revised: 1 as an example, activating the unit controlling y-axis movement with positive value can drive the Ant move left, and a negative value makes the Ant move right.

Original: We will illustrate the approach to probe the correlation in the following subsection.
Revised: We will illustrate the approach to probe the correlation in the following subsection.

Original: Identifying Correlation Coefficient .
Revised: Identifying Correlation Coefficient .

Original: We compute the correlation coefficient through spectral phase discrepancy .
Revised: We compute the correlation coefficient through spectral phase discrepancy .

Original: we first find the predominant frequency component for each signal:
Revised: we first find the predominant frequency component for each signal:

Original: [Equation]
Revised: [Equation]

Original: At each window of STFT, we can compute the phase discrepancy of the transforms of m j and s j at the predominant frequency ω ∗ .
Revised: At each window of STFT, we can compute the phase discrepancy of the transforms of m j and s j at the predominant frequency ω ∗ .

Original: The average phase discrepancy at all temporal windows is computed across N collected episodes:
Revised: The average phase discrepancy at all temporal windows is computed across N collected episodes:

Original: [Equation]
Revised: [Equation]

Original: where D is the number of temporal windows and Φ is the function retrieving the phase.
Revised: where D is the number of temporal windows and Φ is the function retrieving the phase.

Original: We then normalize p j to get correlation coefficient ρ between m j and s j as ρ j = 1 − | 2 p j | /π.
Revised: We then normalize p j to get correlation coefficient ρ between m j and s j as ρ j = 1 − | 2 p j | /π.

Original: ρ j tells us the most important thing that whether the s j will be amplified or reduced by overwriting the output of m j with a positive or negative value.
Revised: ρ j tells us the most important thing that whether the s j will be amplified or reduced by overwriting the output of m j with a positive or negative value.

Original: After that, v j is finally determined by
Revised: After that, v j is finally determined by

Original: [Equation]
Revised: [Equation]

Original: where c j is a magnitude coefficient determining the scale of v j and thus ˙
Revised: where c j is a magnitude coefficient determining the scale of v j and thus ˙

Original: s j .
Revised: s j .

Original: We usually determine the c j by observing the performance after trying evoking the desired behavior with different c j , while it can also be determined by an external PID controller, which takes the error
Revised: We usually determine the c j by observing the performance after trying evoking the desired behavior with different c j , while it can also be determined by an external PID controller, which takes the error

Original: ˆ s − s jt
Revised: ˆ s − s jt

Original: [Figure]
Revised: [Figure]

Original: T r a i n i
Revised: T r a i n i

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: In the Cassie environment, the agent is trained to move forward.
Revised: In the Cassie environment, the agent is trained to move forward.

Original: However, crouching, jumping and sidestepping required to reach
Revised: However, crouching, jumping and sidestepping required to reach

Original:  Experiments
Revised:  Experiments

Original: We first apply Policy Dissection to an autonomous driving task to study the learning dynamics in
Revised: We first apply Policy Dissection to an autonomous driving task to study the learning dynamics in

Original: Sec. 4.2.
Revised: Sec. 4.2.

Original: We demonstrate that the emergent pivotal motor primitives are highly correlated with the success of policy learning.
Revised: We demonstrate that the emergent pivotal motor primitives are highly correlated with the success of policy learning.

Original: With the trained driving policy, we show a novel application of the shared autonomy in Sec. 4.3: improving test-time generalization, where Human-AI shared control can be an alternative to overcoming the generalization gap between training and unseen test-time environment.
Revised: With the trained driving policy, we show a novel application of the shared autonomy in Sec. 4.3: improving test-time generalization, where Human-AI shared control can be an alternative to overcoming the generalization gap between training and unseen test-time environment.

Original: The experiment demonstrates that our method finds not only the correlation but causality between unit output and kinematics.
Revised: The experiment demonstrates that our method finds not only the correlation but causality between unit output and kinematics.

Original: In Sec. 4.4, we conduct experiments on quadrupedal robot and demonstrate that a quadrupedal robot trained to traverse complex terrain with only proprioceptive states can transfer to accomplish the obstacle avoidance task with a bit human effort, even if it does not have any information of the surrounding.
Revised: In Sec. 4.4, we conduct experiments on quadrupedal robot and demonstrate that a quadrupedal robot trained to traverse complex terrain with only proprioceptive states can transfer to accomplish the obstacle avoidance task with a bit human effort, even if it does not have any information of the surrounding.

Original: For qualitatively evaluating our method, we provide a demonstration in Sec. 4.5, where a bipedal Cassie robot trained for moving forward can solve a challenging parkour environment with the help of human subject.
Revised: For qualitatively evaluating our method, we provide a demonstration in Sec. 4.5, where a bipedal Cassie robot trained for moving forward can solve a challenging parkour environment with the help of human subject.

Original: In Sec. 4.6, we investigate the coarseness of the goal-conditioned control enabled by Policy Dissection through a command tracking experiment , and compare it with the state-of-the-art goal-conditioned controller in IsaacGym [42, 57].
Revised: In Sec. 4.6, we investigate the coarseness of the goal-conditioned control enabled by Policy Dissection through a command tracking experiment , and compare it with the state-of-the-art goal-conditioned controller in IsaacGym [42, 57].

Original: 4.1 Experimental Setting
Revised: 4.1 Experimental Setting

Original: The experiments conducted on Cassie and ANYmal in IsaacGym [57], Ant and Walker in MuJoCo [69] and the BipedalWalker [9] follow the general setting.
Revised: The experiments conducted on Cassie and ANYmal in IsaacGym [57], Ant and Walker in MuJoCo [68] and the BipedalWalker [9] follow the general setting.

Original: Other experiments conducted on
Revised: Other experiments conducted on

Original: MetaDrive and Pybullet-A1 environments are introduced as follows:
Revised: MetaDrive and Pybullet-A1 environments are introduced as follows:

Original: MetaDrive .
Revised: MetaDrive .

Original: We train agents to accomplish autonomous driving task in the safety environment of
Revised: We train agents to accomplish autonomous driving task in the safety environment of

Original: MetaDrive [39].
Revised: MetaDrive [39].

Original: In this task, the goal is to maneuver the target vehicle to its destination with low-level continuous control actions.
Revised: In this task, the goal is to maneuver the target vehicle to its destination with low-level continuous control actions.

Original: An episode terminates only when the vehicle arrives at the destination, drives out of road, or collides with the sidewalk.
Revised: An episode terminates only when the vehicle arrives at the destination, drives out of road, or collides with the sidewalk.

Original: The observation of autonomous vehicle consists of map, navigation, lidar, and state information such as speed, heading and side distance.
Revised: The observation of autonomous vehicle consists of map, navigation, lidar, and state information such as speed, heading and side distance.

Original: More training environment details can be found in appendix.
Revised: More training environment details can be found in appendix.

Original: Pybullet-A1 .
Revised: Pybullet-A1 .

Original: The legged robots locomotion experiments are conducted on the Pybullet-UnitreeA1 [16] robot.
Revised: The legged robots locomotion experiments are conducted on the Pybullet-UnitreeA1 [16] robot.

Original: We follow the same environment setting used by Locotransformer robot [76], including terrain shape, obstacle distributions, sensors, reward definition, and termination condition.
Revised: We follow the same environment setting used by Locotransformer robot [75], including terrain shape, obstacle distributions, sensors, reward definition, and termination condition.

Original: Two agents are trained.
Revised: Two agents are trained.

Original: One “insensible” agent with only proprioceptive state is trained to walk on uneven terrain, while the other agent with Locotransformer is directly trained to avoid obstacles.
Revised: One “insensible” agent with only proprioceptive state is trained to walk on uneven terrain, while the other agent with Locotransformer is directly trained to avoid obstacles.

Original: The shared control system will be built on the “insensible” agent, where human subjects help the agent sidestep obstacles.
Revised: The shared control system will be built on the “insensible” agent, where human subjects help the agent sidestep obstacles.

Original: Training Set and Test Set .
Revised: Training Set and Test Set .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4, for both autonomous driving and robot locomotion tasks, we prepare a training set and a held-out test set.
Revised: 4, for both autonomous driving and robot locomotion tasks, we prepare a training set and a held-out test set.

Original: For driving task, the test set is used to benchmark the test-time generalizability of human-AI shared control system and AI-only control system.
Revised: For driving task, the test set is used to benchmark the test-time generalizability of human-AI shared control system and AI-only control system.

Original: We train autonomous driving policies in 50 different environments where the traffic condition
Revised: We train autonomous driving policies in 50 different environments where the traffic condition

Original: [Table]
Revised: [Table]

Original: is mild (averagely 2 cars per map) and all traffic vehicles and the target vehicle can drive smoothly towards their destinations.
Revised: is mild (averagely 2 cars per map) and all traffic vehicles and the target vehicle can drive smoothly towards their destinations.

Original: No obstacles are in these environments.
Revised: No obstacles are in these environments.

Original: In testing time, the trained agents will be evaluated in the other 20 unseen maps with higher traffic density (averagely 6 cars per map).
Revised: In testing time, the trained agents will be evaluated in the other 20 unseen maps with higher traffic density (averagely 6 cars per map).

Original: Besides, obstacles like traffic cones and breakdown vehicles will scatter randomly on the road.
Revised: Besides, obstacles like traffic cones and breakdown vehicles will scatter randomly on the road.

Original: In this case, the target vehicle needs to be maneuvered delicately.
Revised: In this case, the target vehicle needs to be maneuvered delicately.

Original: We compare the performance and safety of the trained policy in two scenarios: runs in the test set solely, and runs with the human-AI shared control system built from Policy Dissection .
Revised: We compare the performance and safety of the trained policy in two scenarios: runs in the test set solely, and runs with the human-AI shared control system built from Policy Dissection .

Original: The primal task for the legged robot is to move toward one direction as fast as possible.
Revised: The primal task for the legged robot is to move toward one direction as fast as possible.

Original: We design the test environment to be a forest-like environment where tree obstacles are randomly scattered.
Revised: We design the test environment to be a forest-like environment where tree obstacles are randomly scattered.

Original: Therefore, though the overall goal is still to move forward, the agent needs extra effort to sidestep obstacles in the test environment.
Revised: Therefore, though the overall goal is still to move forward, the agent needs extra effort to sidestep obstacles in the test environment.

Original: Evaluation Metrics .
Revised: Evaluation Metrics .

Original: For driving task, we evaluate agents on all the 20 test environments, and report the ratio of episodes where the agent arrives at the destination as the success rate .
Revised: For driving task, we evaluate agents on all the 20 test environments, and report the ratio of episodes where the agent arrives at the destination as the success rate .

Original: Also, Out of Road rate can be calculated if the termination is caused by driving out of road.
Revised: Also, Out of Road rate can be calculated if the termination is caused by driving out of road.

Original: For benchmarking the safety of agents, the collision to vehicles, obstacles, sidewalk and buildings raises a cost +1 .
Revised: For benchmarking the safety of agents, the collision to vehicles, obstacles, sidewalk and buildings raises a cost +1 .

Original: In addition, human attention cost is a huge concern of human-AI shared control system.
Revised: In addition, human attention cost is a huge concern of human-AI shared control system.

Original: We quantify this using human involvement , which is measured by the number of environment steps when the human subject presses keys for activating units, or controls throttle, brake, and steering wheel.
Revised: We quantify this using human involvement , which is measured by the number of environment steps when the human subject presses keys for activating units, or controls throttle, brake, and steering wheel.

Original: We average the sum of reward, cost and human involvement on 20 environments as episodic return , episodic cost and human involvement .
Revised: We average the sum of reward, cost and human involvement on 20 environments as episodic return , episodic cost and human involvement .

Original: The above evaluation process will be repeated 5 times for each agent, and we report the average value and std for all metrics.
Revised: The above evaluation process will be repeated 5 times for each agent, and we report the average value and std for all metrics.

Original: For the locomotion task, we use moving distance to measure the performance besides reward, since it directly reflects the goal.
Revised: For the locomotion task, we use moving distance to measure the performance besides reward, since it directly reflects the goal.

Original: Moving distance is the displacement of robot on a specific direction, i.e. x-axis.
Revised: Moving distance is the displacement of robot on a specific direction, i.e. x-axis.

Original: The evaluation is also repeated for 5 times on the exclusive test environment.
Revised: The evaluation is also repeated for 5 times on the exclusive test environment.

Original: Human Interface.
Revised: Human Interface.

Original: We use keyboard as interface in all human-AI shared control experiments.
Revised: We use keyboard as interface in all human-AI shared control experiments.

Original: When a specific key is pressed for evoking a behavior, related motor primitives will be activated and output v j .
Revised: When a specific key is pressed for evoking a behavior, related motor primitives will be activated and output v j .

Original: The behavior termination can be triggered by ways: 1 .
Revised: The behavior termination can be triggered by ways: 1 .

Original: Press another key and change to another behavior 2 .
Revised: Press another key and change to another behavior 2 .

Original: The length of activation period T =
Revised: The length of activation period T =

Original: t 1 − t 0 reaches a predefined length.
Revised: t 1 − t 0 reaches a predefined length.

Original: Once a behavior is terminated, the related motor primitives’ output will not be overwritten.
Revised: Once a behavior is terminated, the related motor primitives’ output will not be overwritten.

Original: In addition, there is one key mapping to the default behavior that no motor primitives are activated.
Revised: In addition, there is one key mapping to the default behavior that no motor primitives are activated.

Original: All experiments without human are repeated 5 times with different random seeds.
Revised: All experiments without human are repeated 5 times with different random seeds.

Original: N = 5 episodes are used for applying Policy Dissection .
Revised: N = 5 episodes are used for applying Policy Dissection .

Original: For experiments involving human subject, we repeat each experiment with 5 human subjects.
Revised: For experiments involving human subject, we repeat each experiment with 5 human subjects.

Original: For driving task, all human subjects have driving license.
Revised: For driving task, all human subjects have driving license.

Original: Human subject can pause the experiments if any discomfort happens.
Revised: Human subject can pause the experiments if any discomfort happens.

Original: No human subjects were harmed in the experiments.
Revised: No human subjects were harmed in the experiments.

Original: Human subjects have signed the experiment agreement and get compensation.
Revised: Human subjects have signed the experiment agreement and get compensation.

Original: 4.2 Policy Dissection for Understanding Learning Dynamics
Revised: 4.2 Policy Dissection for Understanding Learning Dynamics

Original: In this section, we verify the correlation between motor primitives and relevant kinematic attributes.
Revised: In this section, we verify the correlation between motor primitives and relevant kinematic attributes.

Original: We first train PPO [61] agent on the MetaDrive’s training environments and then conduct Policy
Revised: We first train PPO [61] agent on the MetaDrive’s training environments and then conduct Policy

Original: Dissection on the trained policy and find two motor primitives in which Primitive 1 is associated with Speed and Primitive 2 associated with Side Distance .
Revised: Dissection on the trained policy and find two motor primitives in which Primitive 1 is associated with Speed and Primitive 2 associated with Side Distance .

Original: We then plot the evolution of the frequency discrepancy between the identified motor primitives and the kinematics alongside with the macroscopic measure of the learning progress in Fig.
Revised: We then plot the evolution of the frequency discrepancy between the identified motor primitives and the kinematics alongside with the macroscopic measure of the learning progress in Fig.

Original: Note that the frequency discrepancies d
Revised: Note that the frequency discrepancies d

Original: = [ d 1 , ..., d T ] is normalized to [0 , 1] by ˜ d =
Revised: = [ d 1 , ..., d T ] is normalized to [0 , 1] by ˜ d =

Original: [Equation]
Revised: [Equation]

Original: min
Revised: min

Original: [Equation]
Revised: [Equation]

Original: The upper plot in Fig.
Revised: The upper plot in Fig.

Original: 5 shows Primitive 1 gradually emerges to control speed, indicated by the decreasing discrepancy.
Revised: 5 shows Primitive 1 gradually emerges to control speed, indicated by the decreasing discrepancy.

Original: With strong negative correlation, the success rate rapidly increases when the discrepancy of speed primitive also rapidly decreases between epoch 20-30.
Revised: With strong negative correlation, the success rate rapidly increases when the discrepancy of speed primitive also rapidly decreases between epoch 20-30.

Original: Similarly, in the lower plot in Fig. 5, the decreasing discrepancy between Primitive 2 and side distance implies that the internal unit of the agent is gradually specialized to control side distance.
Revised: Similarly, in the lower plot in Fig. 5, the decreasing discrepancy between Primitive 2 and side distance implies that the internal unit of the agent is gradually specialized to control side distance.

Original: There exists a strong correlation between the discrepancy and the out-of-road rate, which can only be reduced by improving the ability to keep side distance.
Revised: There exists a strong correlation between the discrepancy and the out-of-road rate, which can only be reduced by improving the ability to keep side distance.

Original: We thus conclude that the pivotal primitives emerge during training.
Revised: We thus conclude that the pivotal primitives emerge during training.

Original: 4.3 Human-AI Shared Control for Test-time Generalization
Revised: 4.3 Human-AI Shared Control for Test-time Generalization

Original: Using the motor primitives identified above, we can determine the activation value of motor primitive for steering associated kinematics according to Eq. 8.
Revised: Using the motor primitives identified above, we can determine the activation value of motor primitive for steering associated kinematics according to Eq. 8.

Original: We apply the same process to dissect the policy trained from SAC [23] with the same network structure and build a stimulation-evoked map for shared control.
Revised: We apply the same process to dissect the policy trained from SAC [23] with the same network structure and build a stimulation-evoked map for shared control.

Original: We then invite human subject to collaborate with the SAC and PPO agents on the test environments with different takeover methods.
Revised: We then invite human subject to collaborate with the SAC and PPO agents on the test environments with different takeover methods.

Original: As shown in Tab. 1, the average human involvement, episodic cost, and success rate in the test environments are reported across policies with or without human involvement.
Revised: As shown in Tab. 1, the average human involvement, episodic cost, and success rate in the test environments are reported across policies with or without human involvement.

Original: Human involvement greatly improves performance and safety when deploying the policy in unseen environments where delicate maneuvers such as side-passing and yielding are evoked by human, if necessary, to help finish the task.
Revised: Human involvement greatly improves performance and safety when deploying the policy in unseen environments where delicate maneuvers such as side-passing and yielding are evoked by human, if necessary, to help finish the task.

Original: Different from the original PPO policy which is an MLP with 2 hidden layers, 256 units per layer and “tanh” activation function, we also repeat the same experiment for Deep-PPO agent which hashidden layers, and Relu-PPO which has “Relu” as activation function.
Revised: Different from the original PPO policy which is an MLP with 2 hidden layers, 256 units per layer and “tanh” activation function, we also repeat the same experiment for Deep-PPO agent which hashidden layers, and Relu-PPO which has “Relu” as activation function.

Original: The results reported in 1 show our method is invariant to the network architectures and RL algorithms.
Revised: The results reported in 1 show our method is invariant to the network architectures and RL algorithms.

Original: 4.4 Human-AI Shared Control for Task Transfer
Revised: 4.4 Human-AI Shared Control for Task Transfer

Original: [Table]
Revised: [Table]

Original: In addition to the improved test-time generalization, the stimulation-evoked map enables RL agent to accomplish new tasks with the help of human.
Revised: In addition to the improved test-time generalization, the stimulation-evoked map enables RL agent to accomplish new tasks with the help of human.

Original: Recent quadrupedal robots trained by RL exhibit promising ability to walk on bumpy terrain via end-to-end control [36].
Revised: Recent quadrupedal robots trained by RL exhibit promising ability to walk on bumpy terrain via end-to-end control [36].

Original: However, the agent only takes proprioceptive states as input without the observation of surroundings, and thus it can not avoid obstacles on its own.
Revised: However, the agent only takes proprioceptive states as input without the observation of surroundings, and thus it can not avoid obstacles on its own.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4, we train the agent only to walk on uneven terrain but test it in a new environment full of obstacles.
Revised: 4, we train the agent only to walk on uneven terrain but test it in a new environment full of obstacles.

Original: By applying our method, motor primitives related to yaw rate and speed control can be found and serve as the interface for human to steer the “insensible” quadrupedal robot to avoid obstacles.
Revised: By applying our method, motor primitives related to yaw rate and speed control can be found and serve as the interface for human to steer the “insensible” quadrupedal robot to avoid obstacles.

Original: This forms the “State w/oH ” method.
Revised: This forms the “State w/oH ” method.

Original: We also include “State+Vision” method [76], which trains a visuomotor policy for legged robot directly in the test environment, enabling it to sidestep obstacles.
Revised: We also include “State+Vision” method [75], which trains a visuomotor policy for legged robot directly in the test environment, enabling it to sidestep obstacles.

Original: As shown in Table 2, with human involvement, Policy Dissection successfully transforms the “insensible” policy that is not fit to this task and greatly improves the performance, achieving comparable performance to the agent specially designed for this task.
Revised: As shown in Table 2, with human involvement, Policy Dissection successfully transforms the “insensible” policy that is not fit to this task and greatly improves the performance, achieving comparable performance to the agent specially designed for this task.

Original: 4.5 Qualitative Result: Parkour Demo
Revised: 4.5 Qualitative Result: Parkour Demo

Original: As shown in the Fig.
Revised: As shown in the Fig.

Original: 4, the training objective of the bipedal Cassie robot is to move forward.
Revised: 4, the training objective of the bipedal Cassie robot is to move forward.

Original: After training, Policy Dissection can discover several skills in the learned representation, as shown in Fig.
Revised: After training, Policy Dissection can discover several skills in the learned representation, as shown in Fig.

Original: Furthermore, these skills can be triggered and controlled by human subject for solving the challenging parkour environment.
Revised: Furthermore, these skills can be triggered and controlled by human subject for solving the challenging parkour environment.

Original: See the demo video for detail.
Revised: See the demo video for detail.

Original: [Figure]
Revised: [Figure]

Original: 4.6 Experiment on the Control Precision
Revised: 4.6 Experiment on the Control Precision

Original: We conduct an experiment to evaluate the control precision enabled by our method.
Revised: We conduct an experiment to evaluate the control precision enabled by our method.

Original: We train a goal-conditioned policy for ANYmal-C robot in IsaacGym [57].
Revised: We train a goal-conditioned policy for ANYmal-C robot in IsaacGym [57].

Original: The trained robot taking raw rate as input can track a target yaw with a PID controller, and thus serves as the baseline.
Revised: The trained robot taking raw rate as input can track a target yaw with a PID controller, and thus serves as the baseline.

Original: For a fair comparison, we directly dissect this policy and identify the yaw primitive.
Revised: For a fair comparison, we directly dissect this policy and identify the yaw primitive.

Original: Likewise, a PID controller that takes the yaw error as input is used to automatically determine the output v j for yaw primitive, enabling tracking target yaw.
Revised: Likewise, a PID controller that takes the yaw error as input is used to automatically determine the output v j for yaw primitive, enabling tracking target yaw.

Original: Note that for the primitive activation control the explicit heading command in input is set to 0.
Revised: Note that for the primitive activation control the explicit heading command in input is set to 0.

Original: Heading tracking results in Fig.
Revised: Heading tracking results in Fig.

Original: 7 show that the control achieved by primitive activation has comparable precision to the goal-conditioned control (see demo video for more detail).
Revised: 7 show that the control achieved by primitive activation has comparable precision to the goal-conditioned control (see demo video for more detail).

Original:  Conclusion and Discussion
Revised:  Conclusion and Discussion

Original: Inspired by the neuroscience approach to study the motor cortex in primates, we present a simple yet effective method called Policy Dissection to align internal neural representations and kinematic attributes of a pretrained neural policy.
Revised: Inspired by the neuroscience approach to study the motor cortex in primates, we present a simple yet effective method called Policy Dissection to align internal neural representations and kinematic attributes of a pretrained neural policy.

Original: The resulting stimulation-evoked map allows human to steer the agent and evoke its certain behaviors.
Revised: The resulting stimulation-evoked map allows human to steer the agent and evoke its certain behaviors.

Original: Policy Dissection enables human-AI shared control systems on top of the models trained from standard RL algorithms, without any re-training or modification.
Revised: Policy Dissection enables human-AI shared control systems on top of the models trained from standard RL algorithms, without any re-training or modification.

Original: We quantitatively evaluate the human-AI shared control system in autonomous driving and locomotion tasks, and the result suggests the substantial improvement of the test-time generalization for task transfer.
Revised: We quantitatively evaluate the human-AI shared control system in autonomous driving and locomotion tasks, and the result suggests the substantial improvement of the test-time generalization for task transfer.

Original: We further provide interactive demonstrations in various environments to show the generality of our method and its application for human-AI collaboration.
Revised: We further provide interactive demonstrations in various environments to show the generality of our method and its application for human-AI collaboration.

Original: Limitations and societal impact .
Revised: Limitations and societal impact .

Original: One limitation of Policy Dissection is that we only evaluate this method in continuous control tasks with the state vector as input.
Revised: One limitation of Policy Dissection is that we only evaluate this method in continuous control tasks with the state vector as input.

Original: A wider range of tasks and neural network architecture such as discrete control tasks and CNNs can be adopted in future research.
Revised: A wider range of tasks and neural network architecture such as discrete control tasks and CNNs can be adopted in future research.

Original: One potential negative social impact is that the safety issue remains to be solved when deploying shared control systems in real-world applications.
Revised: One potential negative social impact is that the safety issue remains to be solved when deploying shared control systems in real-world applications.

Original: As shown in the autonomous driving experiment, our method derived from empirical study has no theoretical guarantee on the safety of human subjects.
Revised: As shown in the autonomous driving experiment, our method derived from empirical study has no theoretical guarantee on the safety of human subjects.

Original: The unsuccessful collaboration or conflict between human and AI may lead to potential risks.A Code and Demo Video.
Revised: The unsuccessful collaboration or conflict between human and AI may lead to potential risks.

Original: In the project webpage: https://metadriverse.github.io/policydissect , we provide demonstrative videos showing the Policy Dissection for human-AI shared control and the source code including all trained controllers.

Original: B Workflow of Policy Dissection

Original: After identifying the motion generation block, motor primitives , we can pick one or more of them to activate simultaneously and evoke a certain behavior.

Original: Therefore, a group of units are aligned with a certain behavior and stimulation-evoked map can be built.

Original: C Human Subject’s Comments on Shared Control

Original: The performance of shared controls system is largely determined by the effective coordination between human subject and the AI.

Original: Some of our experiment participants report that collaboration with quadrupedal AI is more difficult than with self-driving AI.

Original: The possible reason is that human subjects are more familiar with driving vehicles than controlling legged robots.

Original: For the shared driving task, human subjects have better global planning abilities about which lane to choose and when to stop.

Original: However, for controlling the legged robots on bumpy terrain and avoiding obstacles, some of them didn’t consider how the complex terrain would influence the pose of legged robots and thus had a hard time interacting with the quadrupedal AI.

Original: For example, when the robot is moving down from a small slope and is pitching forward, stopping command should be issued cautiously.

Original: Otherwise, the agent may roll or flip forward.

Original: Therefore, it would be better if human subjects have prior knowledge about the task, so that they can collaborate better with the AI trained for this task.

Original: How to implement effective human-AI teaming in general is an important but less explored direction.

Original: D Summary of Qualitative Results

Original: Besides the results shown in the main paper, we provide more qualitative results of human-AI shared control on policies trained in MetaDrive, Pybullet-A1, Ant, Walker and BipedalWalker environments.

Original: As shown in Fig.

Original: 8, in the top-down view we visualize the trajectories of the PPO agent with or without human shared control on the test environment of driving task.

Original: Since the PPO agent is trained in the mild traffic density without obstacles, it is incompatible to brake and side-pass.

Original: In contrast, with human in the control loop, the lane changing and braking can be deliberately evoked by human, helping the agent solve the near-accidental cases.

Original: We provide a summary on how to identify and use motor primitives to evoke behaviors for the trained agents in different environments.

Original: [Figure]

Original: MetaDrive: Brake.

Original: By activating the motor primitive positively related to speed with a negative value, the agent can gradually decrease the speed and still stay in the same lane.

Original: The brake can be evoked to avoid collision to obstacles cut- vehicles which never

Original: MetaDrive: Lane Changing.

Original: Since the agent can observe the distance to left and right side, identifying motor primitives related to sidewalk and yellow solid line can make the distance to side controllable.

Original: We use “tanh” as activation function which is symmetric.

Original: Therefore identifying only one motor primitive related to one side is enough to evoke lane changing behavior, since, for example, increasing the distance to left side equals decreasing the distance to right side.

Original: Lane changing is useful for sidestepping obstacles or moving to another lane with lower traffic density.

Original: Pybullet-A1:

Original: Turning Left/Right.

Original: As shown in Fig.

Original: 9, the moving direction of legged robot can be controlled by activating the neurons related to yaw or yaw rate.

Original: Similarly, negative and positive stimulation makes the agent turn towards opposite direction.

Original: Pybullet-A1: Stopping.

Original: We can also evoke deceleration or stopping behaviors on the legged robot by controlling the activation of speed-related units.

Original: Since a huge deceleration may make the agent roll forward, we also add a pitch control to suppress the possible front flipping.

Original: Gym-Ant: Stopping. .

Original: The ant will stop if motor primitive associated with x-axis speed is activated.

Original: Gym-Ant: Moving Up/Down .

Original: The ant can move up/down if we activate two primitives respectively related to: 1) Speed of y-axis and 2) heading direction.

Original: Similar to evoking legged robot turning behavior, opposite activation values make the agent move in opposite direction.

Original: Gym-Ant: Spinning .

Original: The ant can spin

Original: if we activate unis related to yaw rate.

Original: Gym-Walker: Deceleration .

Original: The agent trained in Walker stops because we stimulate the motor primitives positively associated with velocity with a negative stimulation.

Original: As a result, the low-level action sequence for deceleration is executed.

Original: Gym-Walker: Hopping .

Original: We manage to find the motor primitives associated with torque force.

Original: We then disable the movement of the knee in red leg by stimulating the motor primitive with negative value.

Original: As a result, the agent hops with only the yellow leg.

Original: Gym-BidepalWalker: Jumping .

Original: For the agent trained in BipedalWalker, jumping is achieved by stimulating motor primitives related to V z , where Z-axis is upward.

Original: Gym-BidepalWalker: Front Flipping .

Original: We train the BidepdalWalker walking with larger torque, and thus it can jump higher if we activate units associated with V z .

Original: BipedalWalker can conduct front flipping, which is the combination of jumping and pitching.

Original: This is resulted from combining another motor primitive related to pitch into control.

Original: Therefore, the walker jumps with increasing angular velocity, performing front flipping.

Original: Gym-BidepalWalker: Standing Up from Split .

Original: In addition, a common failure mode of agent trained in BipedalWalker is that the agent performs split and can not stand up when both legs touch

Original: [Figure]

Original: the ground.

Original: We identify the motor primitives associated with all the motor torques.

Original: Stimulating these motor primitives , the agent manages to stand up and continue moving forward.

Original: IsaacGym-Cassie: Crouching and Tiptoe .

Original: The crouching and tiptoe behaviors are related to the z-height of the main robot body, so that we can identify z-axis related units, and activate them to change the z-height.

Original: IsaacGym-Cassie: Backflip .

Original: This behavior can be described by increasing 1 .

Original: height 2 . pitch and 3 .

Original: knee force.

Original: Therefore, we activate three corresponding motor primitives selected by Eq.

Original: 4 with output calculated by Eq.

Original: 8 to make the robot back flip.

Original: Note that we also manually determine the termination time t 1 for this action, preventing it from continuously back flipping

Original: IsaacGym-Cassie: Jumping .

Original: The jumping is represented by increasing 1 .

Original: knee torque force 2 .

Original: forward speed/moving distance.

Original: Therefore, activating corresponding motor primitives evoke jumping.

Original: Similar as the back-flip, the activation period T = t 1 − t 0 is predetermined, so that it only jump once after pressing the key for jumping.

Original: IsaacGym-Cassie: Redirection .

Original: The redirection can be achieved in the same way as Gym-Ant, activating y-axis movement related units or heading related units.

Original: The forward movement can be evoked by activating x-axis related motor primitives .

Original: [Figure]

Original: E Details and Visualization of Control Precision Experiment

Original: We provide a visualization results of the control precision experiments, as shown in Fig.

Original: It is obvious that both both goal-conditioned control methods can drive ANYmal-C to track the yaw command smoothly and stably.

Original: The main difference is that the goal-conditioned control enabled by

Original: Policy Dissection has a slightly longer response time when changing command, compared to explicit goal-conditioned control.

Original: The demo video is availabe here .

Original: F Environment Details

Original: We provide more environment details for MetaDrive and Pybullet-A1 such as the observations, the design of reward functions, and the termination conditions.

Original: F.1 MetaDrive

Original: In the driving task, the objective of RL agents is to steer the target vehicles with low-level continuous control actions, namely acceleration, brake, and steering.

Original: Observation.

Original: The observation of RL agents is as follows:

Original: • A 240-dimensional vector denoting the Lidar-like point clouds with 50 m maximum detecting distance centering at the target vehicle.

Original: Each entry is in [0 , 1] with Gaussian noise and represents the relative distance of the nearest obstacle in the specified direction.

Original: • A vector containing the data that summarizes the target vehicle’s state such as the steering, heading, velocity, and relative distance to the left and right boundaries.

Original: • The navigation information that guides the target vehicle toward the destination.

Original: We sparsely spread a set of checkpoints, 50m apart on average, in the route and use the relative positions toward future checkpoints as additional observation to the target vehicle.

Original: Reward and Cost Scheme.

Original: The reward function is composed of four parts as follows:

Original: [Equation]

Original: The displacement reward R disp =

Original: d t − d t − 1 , wherein the d t and d t − 1 denotes the longitudinal movement of the target vehicle in Frenet coordinates of current lane between two consecutive time steps, provides dense reward to encourage agent to move forward.

Original: The speed reward R speed = v t /v max incentives agent to drive fast.

Original: v t and v max denote the current velocity and the maximum velocity ( 80 km/h ), respectively.

Original: We also define a sparse terminal reward R term , which is non-zero only at the last time step.

Original: At that step, we set R disp =

Original: [-R speed = 
-]
Revised: [-R speed = 
-]

Original: and assign R term according to the terminal state.

Original: R term is set to +10 if the vehicle reaches the destination, − 5 for crashing others or violating the traffic rule.

Original: We set c 1 = 1 and c 2 = 0 .

Original: For measuring the safety, collision to vehicles, obstacles, sidewalk raises a cost +1 at each time step.

Original: The sum of cost generated in one episode is episode cost, a metric like episode reward, but reflecting safety instead.

Original: Termination Conditions and Evaluation Metrics.

Original: Since we attempt to benchmark the safety of shared-control system, collision to vehicles and obstacles will not terminate the episode.

Original: The episode will be terminated only when: 1) the agent drive out of the drivable area and 2) the agent arrives the destination.

Original: For each trained agent, we evaluate it in 20 held-out test environment and define the ratio of episodes where the agent arrives at the destination as the success rate .

Original: The definition is the same for out of road .

Original: Also, the average episode reward and cost on 20 test environment produce two metrics: Episodic Reward and Episodic Cost .

Original: Since each agent are trained across 5 random seeds, this evaluation process will be executed for 5 agent which has same training setting but different random seeds.

Original: We report the average and std on the 4 metrics mentioned above.

Original: F.2 Pybullet-A

Original: In the quadrupedal locomotion task, the RL training objective is to move forward on bumpy terrain as fast as possible.

Original: We use two environments: the one without obstacles as training environment, and the other one with obstacles as test environment.

Original: The environment construction, reward definition, tasks are the same as [76].

Original: Its environment has open-source code at: https://github.com/Mehooz/ vision4leg .

Original: In this work, we only slightly modify the observation as follows:

Original: • IMU recording Yaw, Yaw rate, Pitch and Roll • Angle of 12 joints • Torque applied to 12 joints

Original: Containing historical output of these sensors over the last 3 steps, the proprioceptive observation is in 84 dimension.

Original: For LocoTransformer baseline method directly trained on test environment, the input is this state vector and first-view depth images in 64 × 64 over the last 4 steps.

Original: F.3 IsaacGym

Original: We use the same setting reported in [57], and the training code is from https://github.com/ leggedrobotics/legged_gym .

Original: G Learning Curves of Trained Agents G.1 MetaDrive

Original: [Figure]

Original: [Figure]

Original: G.2 Pybullet-A1 Legged Robots

Original: [Figure]

Original: G.3 Gym Environment

Original: [Figure]

Original: H Hyper-parameters H.1 MetaDrive

Original: Since SAC is not sensitive to the choice of hyterparameters, the learning rate 1e − 4 is also suitable for

Original: Deep-SAC (6 Layer) and Relu SAC.

Original: For Deep-PPO and Relu-PPO, learning rate should be slightly decreased due to the change of neural network structure.

Original: Also note that the number of hidden units is changed to 128 per layer for Deep PPO with 6 layer.

Original: This is for keeping the total number of network variables approximate to the network used by default PPO (around 250,000 variables).

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: H.2 Gym Environments and Pybullet-A1 Legged Robots

Original: Agents of Gym environments (Walker/Ant/Bidepadel Walker) are trained by SAC and share the same hyperpatrameter setting.

Original: Quadrupedal robots are trained by PPO, and we follow the same codebase and configuration used in [76].

Original: [Table]

Original: [Table]