Original: Human-AI Shared Control viaFrequency-based Policy Dissection
Revised: Human-AI Shared Control via Policy Dissection

Original: Abstract
Revised: Abstract

Original: Human-AI shared control allows human to interact and collaborate with AI to accomplish control tasks in complex environments.
Revised: Human-AI shared control allows human to interact and collaborate with autonomous agents to accomplish control tasks in complex environments.

Original: Previous Reinforcement Learning(RL) methods attempteda goal-conditioned design to achieve human-controllablepolicies at the cost of redesigning the reward function and training paradigm.
Revised: Previous Reinforcement Learning (RL) methods attempted goal-conditioned designs to achieve human-controllable policies at the cost of redesigning the reward function and training paradigm.

Original: Inspired by the neuroscience approach to investigate the motor cortex in primates, wedevelop a simple yet effective frequency-based approach called Policy Dissectionto align the intermediate representation of the learned neural controller with thekinematic attributes of the agent behavior.
Revised: Inspired by the neuroscience approach to investigate the motor cortex in primates, we develop a simple yet effective frequency-based approach called Policy Dissection to align the intermediate representation of the learned neural controller with the kinematic attributes of the agent behavior.

Original: Without modifying the neural controlleror retraining the model, the proposed approach can convert a given RL-trainedpolicy into a human-interactive policy.
Revised: Without modifying the neural controller or retraining the model, the proposed approach can convert a given RL-trained policy into a human-controllable policy.

Original: We evaluate the proposed approach onthe RL tasks of autonomous driving and locomotion.
Revised: We evaluate the proposed approach on many RL tasks such as autonomous driving and locomotion.

Original: The experiments show thathuman-AI shared control  achieved by Policy Dissection in driving task can substantially improve the performance and safety in unseen traffic scenes.
Revised: The experiments show that human-AI shared control system achieved by Policy Dissection in driving task can substantially improve the performance and safety in unseen traffic scenes.

Original: With human inthe loop, the locomotion robots also exhibit versatile controllable motion skills eventhough they are only trained to move forward.
Revised: With human in the inference loop, the locomotion robots also exhibit versatile controllable motion skills even though they are only trained to move forward.

Original: Our results suggest the promisingdirection of implementing human-AI shared autonomy through interpreting thelearned representation of the autonomous agents.
Revised: Our results suggest the promising direction of implementing human-AI shared autonomy through interpreting the learned representation of the autonomous agents.

Original: Code will be made available.
Revised: Code and demo videos are available at https://metadriverse.github.io/policydissect .

Original:  Introduction
Revised:  Introduction

Original: In recent years, autonomous agents trained from deep Reinforcement Learning (RL) have achievedhuge success in a range of applications from robot control [63, 51], autonomous driving [7, 29], tothe power system in smart building [41].
Revised: In recent years, autonomous agents trained from deep Reinforcement Learning (RL) have achieved huge success in a range of applications from robot control [69, 56, 37, 41, 73], autonomous driving [25, 7, 29], to the power system in smart building [44].

Original: Despite the capability to effectively discover feasible policyunder unknown system dynamics in a model-free setting [61], the  lack of generalizability [14] inunseen environments hinders the real-world deployment of learning-based neural policy.
Revised: Despite the capability of discovering feasible policy under unknown system dynamics in a model-free setting [67], the learning-based agents lack sufficient generalizability [14] in unseen environments, which hinders their real-world deployments.

Original: Furthermore,it remains difficult to understand the internal mechanism and the decision making of the neuralnetwork [30, 72], especially when abnormal behaviors happen [10, 71, 46].
Revised: Furthermore, it remains difficult to understand the internal mechanism and the decision making of the neural network [30, 79], especially when abnormal behaviors happen [10, 78, 49].

Original: The lack of transparencyand controllability limits the application of the end-to-end neural controllers in safety-critical systems.
Revised: Thus the absence of generalizability, transparency and controllability limits the application of the end-to-end neural controllers in safety-critical systems.

Original: One potential direction to make the neural controller safer and more trustworthy is incorporatinghuman into the control loop, which we refer to as human-AI shared control .
Revised: One potential direction to make the neural controller safer and more trustworthy is incorporating human into the inference loop, which we refer to as human-AI shared control .

Original: With human involvement,the performance and the safety of the shared autonomy systems can be substantially improved [49, 15,50], even on new tasks and unseen environments.
Revised: With human involvement, the shared autonomy systems can achieve substantial improvement over the performance and safety [54, 15, 55], even on new tasks and unseen environments.

Original: Human-AI shared control has been implementedin various forms, such as enforcing intervention [28, 47] or providing high-level commands [15] tothe AI models.
Revised: Human-AI shared control has been implemented in various forms, such as enforcing intervention [28, 51] or providing high-level commands [15] to the AI models.

Original: Previous works also explore goal-conditioned reinforcement learning  as a form ofhuman-AI shared control, where a high-level goal is fed as an input to the policy during training[55, 2, 15, 25].
Revised: Previous works mainly explore goal-conditioned reinforcement learning (GCRL) as a form of human-AI shared control, where a high-level goal is fed as an input to the policy during training [60, 2, 15, 25, 57].

Original: Thus the behavior of the learned agent can be controlled by varying the inputgoal [44, 36, 73].
Revised: Thus the behavior of the learned agent can be controlled by varying the input goal [47, 38, 80, 37].

Original: However, training goal-conditioned policy requires additional modification of the 
Revised: However, training goal-conditioned policy requires additional modification to the observation design, the reward function and the training

Original: [Figure]
Revised: [Figure]

Revised: Pybullet-A

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: neural controller, algorithm, and the  reward function, while the human control only happens at thegoal level, still lacking  interpretabilityand control of fine-grained movement.
Revised: process. In the case of bipedal robot for example, it is difficult to design reward functions for each behavior like crouching, backflipping, forward jumping. Also, initial state distribution and goal selection criteria should be meticulously designed for a successful GCRL training [50, 57]. One the other hand, the human control only happens at the goal level, still lacking in the interpretability of low-level neural policy.

Original: In this work, we develop a minimalist approach called Policy Dissection , to enable human-AI sharedcontrol on autonomous agents trained on a wide range of tasks.
Revised: In this work, we develop a minimalist approach called Policy Dissection , to enable human-AI shared control on autonomous agents trained on a wide range of tasks.

Original: Policy Dissection does not imposeassumption on the agent’s training scheme, and it requires neither retraining the controller’s networknor modifying the environment.
Revised: Policy Dissection does not impose assumption on the agent’s training scheme like specially-designed reward function, and it requires neither retraining the controller’s network nor modifying the environment.

Original: Policy Dissection achieves human-AI shared control by firstlydissecting the internal representations of the learned policy and aligning them with specific kinematicattributes, and then modulating the activation of the internal representations to elicit the desiredmotor skills.
Revised: Policy Dissection achieves human-AI shared control by firstly dissecting the internal representations of the learned policy and aligning them with specific kinematic attributes, and then modulating the activation of the internal representations to elicit the desired motor skills.

Original: Policy Dissection takes inspiration from neuroscience studies on motor cortex ofprimates: exerting electrical stimulation on different areas of motor cortex can elicit meaningful bodymovements.
Revised: Policy Dissection is inspired by the neuroscience studies on motor cortex of primates, where exerting electrical stimulation on different areas of motor cortex can elicit meaningful body movements [21].

Original: A stimulation-evoked map can be built to reveal the relationship between the evokedbody movements and the motor neurons located in different areas of motor cortex [21, 20, 33].
Revised: Thus, a stimulation-evoked map can be built to reveal the relationship between the evoked body movements and the motor neurons located in different areas of motor cortex [21, 20, 34].

Original: Inorder to replicate the stimulation-evoked map of a  neural controller, the proposed Policy Dissectionfollows a frequency analysis and matching strategy, associating the kinematic attribute and the unitactivation.
Revised: In order to replicate such a stimulation-evoked map inside a controller powered by artificial neural network, the proposed Policy Dissection conducts a frequency analysis and matching strategy for associating the kinematic attribute and the unit activation.

Original: The procedure of building a stimulation-evoked map by Policy Dissection is shown in Fig.
Revised: The procedure of building a stimulation-evoked map by Policy Dissection is illustrated in Fig.

Original: Concretely, we first roll out the given  policy for several episodes. After collecting a set of timeseries of the activities of all units,  kinematics and dynamics, we identify the units whose activitiesalign with the  kinematic and dynamic attributes in a way that there is the lowest frequency discrepancybetween the specific units and the kinematic attributes in the frequency domain.
Revised: Concretely, we first roll out the given well-trained policy for many episodes and collect a set of time series of the activities of all units, and the variation in kinematics and dynamics. Afterwards, for each kinematic attribute we identify one unit whose activity align best with the variation of this kinematic attribute following the principle that there is the lowest frequency discrepancy between both the time series. After finishing the alignment for

Original: The stimulationevoked map between units and attributes is thus built and used to modulate the activation intensity ofidentified units so that the desired motor skills can be elicited.
Revised: [Figure] all kinematic attributes, the identified units are then called motor primitives 1 whose activation can change corresponding kinematic attributes. Since a behavior can be described by the change of one or more kinematic attributes, the stimulation-evoked map can be built for recording the relationship of motor-primitives ’ activation and evoked behaviors.

Revised: For example, the moving up behavior of the Gym-Ant shown in Fig.

Revised: 2 can be described by turning upward (increasing yaw) and increasing upward speed ( v y ), and thus we can activate corresponding two motor primitives to elicit the desired movement.

Original: Therefore, Policy Dissection constructsan interpretable control interface on top of the trained agent for human to interact with and evokecertain behaviors of the agent, thus achieving human-AI shared control.
Revised: In conclusion, Policy Dissection constructs an interpretable control interface on top of the trained agent for human to interact with and evoke certain behaviors of the agent, thus achieving human-AI shared control.

Original: We evaluate the proposed method on several RL agents ranging from locomotion robots to autonomousdriving agents in simulation environments.
Revised: We evaluate the proposed method on several RL agents ranging from locomotion robots to autonomous driving agents in simulation environments.

Original: Experimental results suggest that meaningful motorprimitives emerge in the internal representation.
Revised: Experimental results suggest that meaningful motor primitives emerge in the internal representation.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, novel behaviors can bedeliberately evoked by activating units related to certain motor primitives, even though they are notthe necessary behaviors to solve the primal task.
Revised: 1, novel behaviors can be deliberately evoked by activating units related to certain kinematic attributes, even though they are not the necessary behaviors to solve the primal task.

Original: In the quantitative evaluation, we use the human-AIshared control system enabled by Policy Dissection to improve the generalization in test-time unseenenvironment and achieve  task transfer.
Revised: In the quantitative evaluation, we use the humanAI shared control system enabled by Policy Dissection to improve the generalization in test-time unseen environment and achieve zero-shot task transfer.

Original: In an autonomous driving task, we first train baseline agentsunder mild traffic conditions and evaluate the driving policy on new test environments containingunseen near-accidental scenes.
Revised: In an autonomous driving task, we first train baseline agents under mild traffic conditions and evaluate the driving policy on new test environments containing unseen near-accidental scenes.

Original: Different from the poor performance of the baseline agent in thesenew environments, the human-AI shared control system achieves superior performance and safetyguarantee, with 95% success rate and almost zero cost.
Revised: Different from the poor performance of the baseline agent in these new environments, the human-AI shared control system achieves superior performance and safety guarantee, with 95% success rate and almost zero cost.

Original: In quadrupedal locomotion task, a robot withonly proprioceptive state input can avoid obstacles with shared control, even though its primal task is[Figure] moving forward.
Revised: In quadrupedal locomotion task, a robot with only proprioceptive state input can avoid obstacles with shared control, even though its primal task is moving forward.

Original: These results show that the proposed Policy Dissection not only helps understandthe learned representations of the neural controller, but also brings the intriguing potential applicationof human-AI shared control.
Revised: These results show that the proposed Policy Dissection not only helps understand the learned representations of the neural controller, but also brings the intriguing potential application of human-AI shared control.

Original:  Related Work
Revised:  Related Work

Original: Human-AI Shared Control .
Revised: Human-AI Shared Control .

Original: Existing methods on human-AI shared control can be roughly dividedinto two categories.
Revised: Existing methods on human-AI shared control can be roughly divided into two categories.

Original: The first category is to have human in the control loopto train learningsystems and then execute without human in test time [74].
Revised: The first category is to have human in the training loop and then the welltrained policy can be executed without human in test time [81].

Original: By incorporating human into the training,previous works successfully improve the performance on visualinput control tasks such as Atarigame [1, 53, 65].
Revised: By incorporating human into the training, previous works successfully improve the performance on visual control tasks such as Atari game [1, 58, 71].

Original: Robotic control tasks also benefit from human feedback [28, 40, 59, 47, 66].
Revised: Robotic control tasks also benefit from human feedback [28, 43, 65, 51, 72].

Original: Anothercategory is to have human in both training and test time to accurately accomplish human-assistivetasks.
Revised: The other category is to have human in both training and test time to accurately accomplish humanassistive tasks.

Original: A series of works study this problem based on the Atari Game [49, 54, 8, 31].
Revised: A series of works study this problem based on the Atari Game [54, 59, 8, 31].

Original: In addition,the human-AI shared control system is built on autonomous vehicle [19], robotic arms [27] and inmulti-agent setting [31].
Revised: In addition, the human-AI shared control system is built on autonomous vehicle [19], robotic arms [27] and in multi-agent setting [31].

Original: The reported results on various tasks reflect the effectiveness and efficiencyof training and coordination with human-assistive AI.
Revised: The reported results on various tasks indicate the effectiveness and efficiency of training and coordination with human-assistive AI.

Original: Unlike prior works, our method allows humanto collaborate with AI in test-time without any training-time interventions.
Revised: Unlike prior works, our method allows human to collaborate with AI in test-time, while eschewing the need for any training-time [Figure] human involvement.

Original: In experiments, sharedcontrol systems are implemented on several robotic control tasks to show the effectiveness of ourmethod.
Revised: In experiments, shared control systems are implemented on several robotic control tasks to show the effectiveness of our method.

Original: Neural Network Interpretability .
Revised: Neural Network Interpretability .

Original: There is a growing interest in the AI community to understandwhat is learned inside the neural network.
Revised: There is a growing interest in the AI community to understand what is learned inside the neural network.

Original: Previous interpretability methods mostly focus on explaining the networks trained for visual tasks [75, 57, 11, 76, 38].
Revised: Previous interpretability methods mostly focus on explaining the networks trained for visual tasks [82, 63, 11, 83, 40].

Original: Various empirical studies have foundthat meaningful and interpretable structures emerge in the internal representation when trained forthe primal tasks.
Revised: Various empirical studies have found that meaningful and interpretable structures emerge in the internal representation when trained for the primal tasks.

Original: For example, object detectors emerge in scene classifier [6], and disentangled layoutand attribute controllers emerge from image generation model [68].
Revised: For example, object detectors emerge in scene classifier [6], and disentangled layout and attribute controllers emerge from image generation model [75].

Original: Varying the disentangled hiddenfeatures of the deep generative model can edit the content of the generated images.
Revised: Varying the disentangled hidden features of the deep generative model can edit the content of the generated images.

Original: On theother hand, visuomotor control tasks can also acquire visual understanding to explain the behaviorchosen by agent with visual input through saliency map [60, 3, 48, 12, 26, 22].
Revised: On the other hand, visuomotor control tasks can also acquire visual understanding to explain the behavior chosen by agent with visual input through saliency map [66, 3, 53, 12, 26, 22].

Original: Neuroscience Inspired Control .
Revised: Neuroscience Inspired Control .

Original: The complex behaviors shown by recent autonomous agents promptresearchers to develop new analysis methods.
Revised: The complex behaviors shown by recent autonomous agents prompt researchers to develop new analysis methods.

Original: As a source of inspiration, neuroscience providesuseful tools [45, 33], and effective procedures [24, 18, 43] for understanding animal brains which arealso like black-box systems.
Revised: As a source of inspiration, neuroscience provides useful tools [48, 34], and effective procedures [24, 18, 46] for understanding animal brains which are also like black-box systems.

Original: Recent works try to develop neural network-based policies from theperspective of neuroscience [17, 4, 70, 13].
Revised: Recent works try to develop neural network-based policies from the perspective of neuroscience [17, 4, 77, 13].

Original: In a closely related work on understanding learned agent,
Revised: In a closely related work on understanding learned agent,

Original: Merel et al.
Revised: Merel et al.

Original: [43] use RSA [34] with CKA [32] similarity index to reveal that behaviors spanningmultiple timescales are encoded in distinct neural layers.
Revised: [46] use RSA [35] with CKA [33] similarity index to reveal that behaviors spanning multiple timescales are encoded in distinct neural layers.

Original: Apart from providing the neuroscientifictools, many concepts from neuroscience, such as motor primitive also inspire researchers to designgeneralizable controllers [64, 62, 42].
Revised: Apart from providing the neuroscientific tools, many concepts from neuroscience, such as motor primitive also inspire researchers to design generalizable controllers [70, 68, 45].

Original:  Policy Dissection Method
Revised:  Policy Dissection Method

Original: Policy Dissection aims at deriving  human-controllable policy from a well-trained RL agent.
Revised: Policy Dissection aims at deriving a human-controllable policy from a well-trained RL agent.

Original: It hastwo unique features to enable human-AI shared control: First, it operates on a well-trained RL agent[Figure] without re-training or modifying the original reward/environment and the policy architecture.
Revised: It has two unique features to enable human-AI shared control: First, it operates on a well-trained RL agent without re-training or modifying the original reward/environment and the policy architecture.

Original: Weonly need to roll out the agent for several episodes and log its neural activity and  kinematic attributes.
Revised: We only need to roll out the agent for several episodes and log its neural activity and the change of kinematic attributes.

Original: Based on the collected records, certain units and kinematic attributes will be aligned.
Revised: Based on the collected data, kinematic attributes are aligned with certain units which are further termed as motor primitives .

Original: Thus ourmethod is non-intrusive to the primal task and trained model, and applicable to a wide range ofenvironments.
Revised: Thus our method is non-intrusive to the primal task and trained model, and applicable to a wide range of environments.

Original: Second, Policy Dissection can discover generalizable motor skills that even the agentis not trained to perform.
Revised: Second, activating certain motor primitives can elicit generalizable motor skills that even the agent is not trained to perform.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, the Ant robot can be steered to move up/down even itis only trained to move forwardas fast as possible in the primal task.
Revised: 1, the bipedal Cassie robot can be steered to tip toe, crouch and back flip even if it is only trained to move forward in the primal task.

Original: We will introduce the threeessential steps of the Policy Dissection method in detail.
Revised: In the following subsections, we will introduce the four essential steps of the Policy Dissection method in detail.

Original: 3.1 Monitoring Neural Activity and Kinematics
Revised: 3.1 Monitoring Neural Activity and Kinematics

Original: Since multi-layer perceptron (MLP) is the most common network architecture used in many RL tasks,we assume the neural controller is a MLP, which has L hidden layers and all layers have the samenumber of I hidden units.
Revised: Since multi-layer perceptron (MLP) is the most common network architecture used in many RL tasks, we assume the neural controller is an MLP with Z units, which has L hidden layers and all layers have the same number of I hidden units 2 .

Original: We denote a neuron by l, i as z l,i , where l = 1 , ..., L is the index of layerand i = 1 , ..., I is the neuron index in each layer, and its output at time step t is z l,it . 
Revised: We denote a neuron by l, i as z l, i ∈ Z , where l = 1 , ..., L is the index of layer and i = 1 , ..., I is the neuron index in each layer, and its output at time step t is z l,it . Let J be the total number of kinematic attributes.

Original: The agent’skinematic attributes are represented by  s jt , j = 1 , ..., J , where J is the total number of attributes.
Revised: The agent’s kinematic attributes are represented by S = { s j }, j = 1 , ..., J , and the measurement of each kinematic attribute s j at time step t is s jt .

Original: We define the motor primitive as the unit m j ∈ { z } that  steers the agent and causes transition ofthe kinematics.
Revised: We define the motor primitive as the unit m j ∈ Z that can steers the agent and cause change of the corresponding kinematic attribute.

Original: We assume that each kinematic attribute s j has a corresponding motor primitive m jto drive the attribute to a desired state, and thus there are also J motor primitives.
Revised: We assume that each kinematic attribute s j has a corresponding motor primitive m j to drive the attribute to a desired state, and thus there are also J motor primitives , denoted by M = { m j } , j = 1 , ..., J .

Original: Our goal is todiscover motor primitive m j by associating the units with certain kinematic attribute.
Revised: Our goal is to discover the motor primitive m j by associating the units with certain kinematic attribute s j .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, we take the Ant robot as anillustrative example.
Revised: 3, we take the Ant robot as an example.

Original: Policy Dissection starts byrolling out the pretrained agent for several episodes and tracking the activation of all units as well asthe kinematics, such as yaw rate and velocity.
Revised: Policy Dissection starts by rolling out the pretrained agent for several episodes and tracking the activation of all units as well as the kinematics, such as yaw rate and velocity.

Original: We record the I · L time series { z l,i =
Revised: We record the I · L time series { z l,i =

Original: [ z l,i 1 , z l,i 2 , ... ] } I · Lmeasuring the neural activities and J time series of kinematic attributes { s j =
Revised: [ z l,i 1 , z l,i 2 , ... ] } I · L measuring the neural activities and J time series of kinematic attributes { s j =

Original: [ s j 1 , s j 2 , ... ] } J forfurther analysis.
Revised: [ s j 1 , s j 2 , ... ] } J for further analysis.

Original: 3.2 Associating Units with Kinematic Attributes
Revised: 3.2 Associating Units with Kinematic Attributes

Original: Observing the neural activity and kinematics recording in Fig.
Revised: Observing the neural activity and kinematic measurement recorded in Fig.

Original: 3, we can find that several repeatedpatterns of changing kinematics appear in different frequencies when executing the policy, whichis reminiscent of animal behavioral research [21, 20, 33].
Revised: 3, we can find that several repeated kinematic patterns appear in different frequencies when executing the policy, which is reminiscent of animal behavioral research [21, 20, 34].

Original: It suggests that the motor skills has itspredominant frequency component [43], controlling fast kinematics and slow kinematics.
Revised: It suggests that the motor skills has its predominant frequency component [46], controlling fast kinematics and slow kinematics.

Original: Sinceneural activation signals also encode distinct frequency information, this resemblance inspires usto process time series ofboth neural activity and kinematic attributes in frequency domain, wherethe difference of predominant frequency between two signals can be clearly revealed and the phase discrepancy of two time series are eliminated.
Revised: Since neural activation signals also encode distinct frequency information as shown in Fig. 3, this resemblance inspires us to make the following hypothesize: For one kinematic attribute, the unit with minimal frequency discrepancy is the motor primitive . To this end, we process time series of neural activity and kinematic attributes in frequency domain, where the difference of predominant frequency between two signals can be clearly revealed and the phase discrepancy of two time series are eliminated.

Original: As a consequence, this allows us to  associate kinematicattributes with units whose activity best matches the corresponding motor primitives.
Revised: As a consequence, this allows us to analyze the recorded data and associate kinematic attributes with units whose activity best matches the variation of target kinematic attributes.

Original: In Policy Dissection, all time series  are transformed into the frequency domain through Discrete-time
Revised: Concretely, all time series { x } including the neural activities { z l,i } and change of kinematic attributes { s j } are transformed into the frequency domain through Discrete-time

Original: Short-time Fourier transform (STFT):
Revised: Short-time Fourier transform (STFT):

Original: [Equation]
Revised: [Equation]

Original: STFT splits the time series x = [ x 1 , x 2 , ... ] into a sequence of temporal window and d ∈ [1 , + ∞ ]denotes the time step of the center of the temporal window.
Revised: STFT splits the time series x = [ x 1 , x 2 , ... ] into a sequence of temporal window and d ∈ [1 , + ∞ ] denotes the time step of the center of the temporal window.

Original: STFT then conducts Fourier transform ineach temporal window.
Revised: STFT then conducts Fourier transform in each temporal window.

Original: ω ∈ [0 , + ∞ ] denotes the frequency.
Revised: ω ∈ [0 , + ∞ ] denotes the frequency.

Original: To reduce spectral leakage and reservethe frequency information as much as possible, the window function W ( t ) used in Policy Dissectionis blackman window .
Revised: To reduce spectral leakage and reserve the frequency information as much as possible, the window function W ( t ) used in Policy Dissection is blackman window .

Original: Moreover, the window length is 64 with hop length 16 .
Revised: Moreover, the window length is 64 with hop length 16 .

Original: We apply STFT insteadof naive Fourier transform since it can retain temporal information by breaking the time series intoseveral overlapping chunks of frames.
Revised: We apply STFT instead of naive Fourier transform since it can retain temporal information by breaking the time series into several overlapping chunks of frames.

Original: This feature is important since in the following frequencymatching we focus on the similarity of resulting spectrograms as well as the temporal correlation oftwo signals.
Revised: This feature is important since in the following frequency matching we focus on the similarity of resulting spectrograms as well as the temporal correlation of two signals.

Original: We use the spectrogram to characterize a measurement:
Revised: We use the spectrogram to characterize a measurement:

Original: [Equation]
Revised: [Equation]

Original: After obtaining the spectrograms for all kinematic attributes and neural activities, frequency discrepancy can be computed for each neuron-kinematics pair ( z l,i , s j ) :[Equation] [Equation] [Figure]
Revised: After obtaining the spectrograms for all kinematic attributes and neural activities, frequency discrepancy can be computed for each neuron-kinematics pair ( z l,i , s j ) :

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: For each kinematic attribute, frequency matching isapplied to select the unit with minimal average discrepancy  as the motor primitive responsible for thevariation of kinematics:
Revised: For each kinematic attribute, frequency matching is applied to select the unit with minimal average discrepancy across N collected episodes as the motor primitive responsible for triggering the change of desired kinematic attributes:

Original: [Equation]
Revised: [Equation]

Revised: 3.3 Building Stimulation-evoked Map

Original:  where N is the number of collected episodes.
Revised: A behavior b k can be described by changing a subset of kinematic attributes S k ⊆ S , which can be achieved by activating a set of corresponding motor primitives M k ⊆ M . Therefore, these movement generation building blocks, motor primitives , are associated with a certain behavior, inducing the stimulation-evoked map { ( M k , b k ) } , k = 1 , ..., K , where K is the number of discovered behaviors. Taking the Cassie’s back-flip shown in Fig.

Original: Webuild { ( m j , s j ) } , j = 1, ..., J as the stimulationevoked map, which further enables human to evokecertain behaviors by activating one or more motorprimitives.
Revised: 1 as an example, this behavior can be described by increasing 1 . height 2 . pitch and 3 . knee force. Therefore, we activate three corresponding motor primitives selected by Eq.

Revised: 4 with output calculated by Eq.

Revised: 8 to make the robot back flip.

Revised: In practice, it is possible that more motor primitives are required.

Revised: Again, taking the back-flip as an example, the force for both knee may be different after activating corresponding units, and thus giving the robot a roll velocity, which can be further eliminated by activating roll related motor primitive .

Original: 3.3 Steering Agent via Stimulation-evoked Map
Revised: 3.4 Steering Agent via Stimulation-evoked Map

Original: Human-AI shared can be fulfilled via the stimulation-evoked map.
Revised: Now the last thing for evoking a behavior b k is to determine the output value for motor primitives S k . At time step t , activating the motor primitive m j related to kinematic attribute s j stands for applying its derivative ˙

Original: As shown in Fig.
Revised: s j to the agent.

Original: 2, we stimulatecertain motor primitives that are tightly coupled with certain kinematics such as spinning (increasingyaw rate) and deceleration (decreasing linear velocity).
Revised: For example, activating the speed related units equals to apply the derivative of speed, acceleration, to the agent. Therefore, changing the kinematic attribute s j at time step t 0 by activating the motor primitive m j with a unit output v j can be formulated as: [Equation]

Original: In this section, we discuss the implementationdetails of the process that elicits the desired behaviors.
Revised: where T = t 1 − t 0 is the activation period of the motor primitive m j

Original: Identifying Correlation Coefficient . The correlation between activation value of  motor primitivesand the kinematics attributes needs to be determined.
Revised: , constant v j is the time-invariant output of m j and f ( v ) is a mapping from motor primitive output value v j to ˙ s j . For steering the agent, our goal is to understand some properties of f ( v ) , especially the correlation between f ( v ) and v , so that we can amplify or reduce the s j by choosing a v or − v .

Original: As shown in Ant robot of Fig.
Revised: Taking the Ant robot of Fig.

Original: 1, activating theneurons controlling y-axis movement with positive value can drive the Ant move left, and a negativevalue makes the Ant move right. 
Revised: 1 as an example, activating the unit controlling y-axis movement with positive value can drive the Ant move left, and a negative value makes the Ant move right. We will illustrate the approach to probe the correlation in the following subsection.

Original:  We compute the correlation coefficient through spectral phase discrepancy .
Revised: Identifying Correlation Coefficient . We compute the correlation coefficient through spectral phase discrepancy .

Original: we first find the predominant frequency component for each signal:
Revised: we first find the predominant frequency component for each signal:

Original: [Equation]
Revised: [Equation]

Original: At each window of STFT, we can compute the phase discrepancy of the transforms of m j and s j atthe predominant frequency ω ∗ .
Revised: At each window of STFT, we can compute the phase discrepancy of the transforms of m j and s j at the predominant frequency ω ∗ .

Original: The average phase discrepancy at all temporal windows is computedacross N collected episodes:
Revised: The average phase discrepancy at all temporal windows is computed across N collected episodes:

Original: [Equation]
Revised: [Equation]

Original: where D is the number of temporal windows and Φ is the function retrieving the phase.
Revised: where D is the number of temporal windows and Φ is the function retrieving the phase.

Original: We thennormalize p j to get correlation coefficient ρ between m j and s j as [Equation]
Revised: We then normalize p j to get correlation coefficient ρ between m j and s j as ρ j = 1 − | 2 p j | /π.

Original: A positive ρ j indicates we can amplify thekinematic attribute s j  by stimulating the motor primitivem j with a positive  value.
Revised: ρ j tells us the most important thing that whether the s j will be amplified or reduced by overwriting the output of m j with a positive or negative value.

Original: Identifying Unit Output . Suppose we want to amplify the scale of kinematic attribute s j , the outputvalue v j for motor primitive m j , namely the activation of m j , can be determined by:
Revised: After that, v j is finally determined by

Original: [Equation]
Revised: [Equation]

Original: where k j is a custom constant decided manually for each motor primitive.
Revised: where c j is a magnitude coefficient determining the scale of v j and thus ˙

Original: Since changing kinematicattribute takes some time, the change of scale will be determined by the length of intervention time. For example, if we want to decrease the speed of an autonomous vehicle by activating speed primitive,the longer the activation time, the larger the speed will be decreased.
Revised: s j .

Original: In the whole activation period,the output of m j will be overwritten with v j . Combining Motor Primitives . Some behavior requires stimulating one or more motor primitives toevoke.
Revised: We usually determine the c j by observing the performance after trying evoking the desired behavior with different c j , while it can also be determined by an external PID controller, which takes the error ˆ s − s jt between target state ˆ s and state s jt c j automatically.

Original: Taking the Ant robot shown in Fig.
Revised: [Figure] T r a i n i ng

Original: 2 as the example, moving towards y-axis can be implemented by two kinematic attributes including increasing V y and heading towards y-axis. Therefore,we activate two corresponding motor primitives selected by Eq.
Revised: T es t

Original: 4 with value calculated by Eq.
Revised: [Figure]

Original: 8 tomake the agentmove towards y-axis. The overall policy dissection workflow is in Appendix.
Revised: In the Cassie environment, the agent is trained to move forward. However, crouching, jumping and sidestepping are required to reach the destination

Original:  Experiments
Revised:  Experiments

Original: We first apply policy dissection to an autonomous driving task to study the learning dynamics in
Revised: We first apply Policy Dissection to an autonomous driving task to study the learning dynamics in

Original: Sec. 4.2.
Revised: Sec. 4.2.

Original: We demonstrate that the emergent pivotal motor primitives are highly correlated withthe success of policy learning.
Revised: We demonstrate that the emergent pivotal motor primitives are highly correlated with the success of policy learning.

Original: In Sec. 4.3, we show a novel application of the shared autonomy:improving the test-time generalization, where Human-AI shared control can be an alternative toovercoming the generalization gap between training and unseen test-time environment.
Revised: With the trained driving policy, we show a novel application of the shared autonomy in Sec. 4.3: improving the test-time generalization, where Human-AI shared control can be an alternative to overcoming the generalization gap between training and unseen test-time environment.

Original: The experiment demonstrates that our method finds not only the correlation but causality between unit outputand kinematics.
Revised: The experiment demonstrates that our method finds not only the correlation but causality between unit output and kinematics.

Original: In Sec. 4.4, we conduct experiments on quadrupedal robot and demonstrate that aquadrupedal robot trained to traverse complex terrain with only proprioceptive state can transfer toaccomplish obstacle avoidance task with a bit human effort, even  it does not have any informationof the surrounding. In sec.
Revised: In Sec. 4.4, we conduct experiments on quadrupedal robot and demonstrate that a quadrupedal robot trained to traverse complex terrain with only proprioceptive state can transfer to accomplish obstacle avoidance task with a bit human effort, even if it does not have any information of the surrounding. For qualitatively evaluting our method, we provide a demonstration in Sec. 4.5, where a bipedal Cassie robot trained for moving forward can solve a challenging parkour environment with the help of human subject.

Original: 4.5, we investigate the command tracking accuracy of  Policy Dissection,and compare it with the state-of-the-art goal-conditioned controller in IsaacGym [39, 52].
Revised: In Sec. 4.6, we investigate the coarseness of the goal-conditioned control enabled by Policy Dissection through a command tracking experiment , and compare it with the state-of-the-art goal-conditioned controller in IsaacGym [42, 57].

Original: Lastly,more qualitative demonstrations in appendix show that our method can be generalized to variousenvironments and autonomy tasks.

Original: 4.1 Experimental Setting
Revised: 4.1 Experimental Setting

Original: Thequalitative experimentsare conducted on the Ant and Walker in MuJoCo [63] and the BipedalWalker [9] following the general setting.
Revised: The experiments conducted on Cassie and ANYmal in IsaacGym [57], Ant and Walker in MuJoCo [69] and the BipedalWalker [9] follow the general setting.

Original: The quantitative experimentsof human-AI shared controlare conducted on MetaDrive and Pybullet-AI environments  introduced as follows:
Revised: Other experiments conducted on MetaDrive and Pybullet-A1 environments are introduced as follows:

Original: MetaDrive .
Revised: MetaDrive .

Original: We train agents to accomplish autonomous driving task in the safety environment of
Revised: We train agents to accomplish autonomous driving task in the safety environment of

Original: MetaDrive [37].
Revised: MetaDrive [39].

Original: In this task, the goal is to maneuver the target vehicle to destination with low-levelcontinuous control actions.
Revised: In this task, the goal is to maneuver the target vehicle to destination with low-level continuous control actions.

Original: An episode terminates only when the vehicle arrives at the destination,drives out of road, or collides with the sidewalk.
Revised: An episode terminates only when the vehicle arrives at the destination, drives out of road, or collides with the sidewalk.

Original: The observation of autonomous vehicle consists ofmap, navigation, lidar and state information such as speed and heading.
Revised: The observation of autonomous vehicle consists of map, navigation, lidar, and state information such as speed, heading and side distance. More training environment details can be found in appendix.

Original: Pybullet-A1 .
Revised: Pybullet-A1 .

Original: The legged robots locomotion experiments are conducted on the Pybullet Unitree-  A1 [
Revised: The legged robots locomotion experiments are conducted on the Pybullet-UnitreeA1 [16] robot.

Original: ] robot We follow the same environment setting used by Locotransformer robot [69], including terrain shape, obstacle distributions, sensors, reward definition, and termination condition.
Revised: We follow the same environment setting used by Locotransformer robot [76], including terrain shape, obstacle distributions, sensors, reward definition, and termination condition.

Original: Two agentsare trained.
Revised: Two agents are trained.

Original: One “insensible” agent with only proprioceptive state is trained to walk on uneven terrain,while the other agent with Locotransformer is directly trained to avoid obstacles.
Revised: One “insensible” agent with only proprioceptive state is trained to walk on uneven terrain, while the other agent with Locotransformer is directly trained to avoid obstacles.

Original: The shared controlsystem will be built on the “insensible” agent, where human subjects help the agent sidestep obstacles.T r a i n i ng E n v i r on m e n t T es t E n v i r on m e n t MetaDrive SafetyEnv
Revised: The shared control system will be built on the “insensible” agent, where human subjects help the agent sidestep obstacles.

Original: Pybullet-A

Original: [Figure]

Original: Training Set and Test Set .
Revised: Training Set and Test Set .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5, for both autonomous driving and robot locomotiontasks, we prepare training set and held-out test set.
Revised: 4, for both autonomous driving and robot locomotion tasks, we prepare training set and held-out test set.

Original: For driving task, the test set is used to benchmarkthe test-time generalizability of human-AI shared control system and AI-only control system.
Revised: For driving task, the test set is used to benchmark the test-time generalizability of human-AI shared control system and AI-only control system.

Original: Wetrain autonomous driving policies in 50 different environments where the traffic condition is mild(averagely 2 cars per map) and all traffic vehicles and the target vehicle can drive smoothly towardstheir destinations.
Revised: We train autonomous driving policies in 50 different environments where the traffic condition is mild [Table] (averagely 2 cars per map) and all traffic vehicles and the target vehicle can drive smoothly towards their destinations.

Original: No obstacles are in these environments.
Revised: No obstacles are in these environments.

Original: In testing time, the trained agents will beevaluated in the other 20 unseen maps with higher traffic density (averagely 6 cars per map).
Revised: In testing time, the trained agents will be evaluated in the other 20 unseen maps with higher traffic density (averagely 6 cars per map).

Original: Besides,obstacles like traffic cones and breakdown vehicles will scatter randomly on the road.
Revised: Besides, obstacles like traffic cones and breakdown vehicles will scatter randomly on the road.

Original: In this case,the target vehicle needs to be maneuvered delicately.
Revised: In this case, the target vehicle needs to be maneuvered delicately.

Original: We compare the performance and safety of thetrained policy in two scenarios: runs in the test set solely, and runs with the human-AI shared controlsystem built from Policy Dissection.
Revised: We compare the performance and safety of the trained policy in two scenarios: runs in the test set solely, and runs with the human-AI shared control system built from Policy Dissection .

Original: The primal task for the legged robot is to move toward one direction as fast as possible.
Revised: The primal task for the legged robot is to move toward one direction as fast as possible.

Original: We designthe test environment to be a forest-like environment where tree obstacles are randomly scattered.
Revised: We design the test environment to be a forest-like environment where tree obstacles are randomly scattered.

Original: Therefore, though the overall goal is still to move forward, the agent needs extra effort to sidestepobstacles in the test environment.
Revised: Therefore, though the overall goal is still to move forward, the agent needs extra effort to sidestep obstacles in the test environment.

Original: Evaluation Metrics .
Revised: Evaluation Metrics .

Original: For driving task, we evaluate agents on all the 20 test environments, and reportthe ratio of episodes where the agent arrives at the destination as the success rate .
Revised: For driving task, we evaluate agents on all the 20 test environments, and report the ratio of episodes where the agent arrives at the destination as the success rate .

Original: Also, Out of Roadrate can be calculated if the termination is caused by driving out of road.
Revised: Also, Out of Road rate can be calculated if the termination is caused by driving out of road.

Original: For benchmarking the safetyof agents, the collision to vehicles, obstacles, sidewalk and buildings raises a cost +1 .
Revised: For benchmarking the safety of agents, the collision to vehicles, obstacles, sidewalk and buildings raises a cost +1 .

Original: In addition,human attention cost is a huge concern of human-AI shared control system.
Revised: In addition, human attention cost is a huge concern of human-AI shared control system.

Original: We quantify this usinghuman involvement , which is measured by the number of environment steps when the human subjectpresses keys for activating units, or controls throttle, brake, and steering wheel.
Revised: We quantify this using human involvement , which is measured by the number of environment steps when the human subject presses keys for activating units, or controls throttle, brake, and steering wheel.

Original: We average the sumof reward, cost and human involvement on 20 environments as episodic return , episodic cost andhuman involvement .
Revised: We average the sum of reward, cost and human involvement on 20 environments as episodic return , episodic cost and human involvement .

Original: The above evaluation process will be repeated 5 times for each agent, and wereport the average value and std for all metrics.
Revised: The above evaluation process will be repeated 5 times for each agent, and we report the average value and std for all metrics.

Original: For the locomotion task, we use moving distanceto measure the performance.
Revised: For the locomotion task, we use moving distance to measure the performance besides reward, since it directly reflects the goal.

Original: Moving distance is the displacement of robot on specific direction, i.e.x-axis.
Revised: Moving distance is the displacement of robot on specific direction, i.e. x-axis.

Original: The evaluation is also repeated for 5 times on the exclusive test environment.
Revised: The evaluation is also repeated for 5 times on the exclusive test environment.

Original: Human Interface.
Revised: Human Interface.

Original: We use keyboard as interface in all human-AI shared control experiments.
Revised: We use keyboard as interface in all human-AI shared control experiments.

Original: When aspecific key is pressed, related motor primitives will be activated with output v j constantly accordingto Eq. 8 until another key is pressed, which will trigger new motor primitives.
Revised: When a specific key is pressed for evoking a behavuir, related motor primitives will be activated and output v j . The behavior termination can be triggered by ways: 1 . Press another key and change to another behavior 2 . The length of activation period T =

Original: Besides, one key mapsto none of the  motor primitives which is pressed torecover the default policy and the natural state.
Revised: t 1 − t 0 reaches a predefined length. Once a behavior is terminated, the related motor primitives’ output will not be overwritten. In addition, there is one key mapping to the default behavior that no motor primitives are activated.

Original: All experiments without human are repeated 5 times with different random seeds.
Revised: All experiments without human are repeated 5 times with different random seeds.

Original: N = 5 episodesare used for applying Policy Dissection.
Revised: N = 5 episodes are used for applying Policy Dissection .

Original: For experiments involving human subject, we repeat eachexperiment with 5 human subjects.
Revised: For experiments involving human subject, we repeat each experiment with 5 human subjects.

Original: For driving task, all human subjects have driving license.
Revised: For driving task, all human subjects have driving license.

Original: Humansubject can pause the experiments if any discomfort happens.
Revised: Human subject can pause the experiments if any discomfort happens.

Original: No human subjects were harmed in theexperiments.
Revised: No human subjects were harmed in the experiments.

Original: Human subjects have signed the experiment agreement and get compensation.
Revised: Human subjects have signed the experiment agreement and get compensation.

Original: [Table]

Original: 4.2 Policy Dissection for Understanding Learning Dynamics
Revised: 4.2 Policy Dissection for Understanding Learning Dynamics

Original: In this section, we verify the correlation between motor primitives and relevant kinematic attributes.
Revised: In this section, we verify the correlation between motor primitives and relevant kinematic attributes.

Original: We first train PPO [56] agent on the MetaDrive’s training environments and then conduct Policy
Revised: We first train PPO [61] agent on the MetaDrive’s training environments and then conduct Policy

Original: Dissection on the trained policy and find two motor primitives in which Primitive 1 is associatedwith Speed and Primitive 2 associated with Side Distance .
Revised: Dissection on the trained policy and find two motor primitives in which Primitive 1 is associated with Speed and Primitive 2 associated with Side Distance .

Original: We then plot the evolution of thefrequency discrepancy between the identified motor primitives and the kinematics alongside withthe macroscopic measure of the learning progress in Fig.
Revised: We then plot the evolution of the frequency discrepancy between the identified motor primitives and the kinematics alongside with the macroscopic measure of the learning progress in Fig.

Original: Note that the frequency discrepanciesd =
Revised: Note that the frequency discrepancies d

Original:  [ d 1 , ..., d T ] is normalized to [0 , 1] by ˜ d 
Revised: = [ d 1 , ..., d T ] is normalized to [0 , 1] by ˜ d = [Equation]

Original: = d − min( d ) max( d ) − min( d ) .

Original: The upper plot in Fig.
Revised: The upper plot in Fig.

Original: 6 shows Primitive 1 gradually emerges to control speed, indicated by thedecreasing discrepancy.
Revised: 5 shows Primitive 1 gradually emerges to control speed, indicated by the decreasing discrepancy.

Original: With strong negative correlation, the success rate rapidly increases whenthe discrepancy of speed primitive also rapidly decreases between epoch 20-30.
Revised: With strong negative correlation, the success rate rapidly increases when the discrepancy of speed primitive also rapidly decreases between epoch 20-30.

Original: Similarly, in thelower plot in Fig. 6, the decreasing discrepancy between Primitive 2 and side distance indicatesthat the internal unit of the agent is gradually specialized to control side distance.
Revised: Similarly, in the lower plot in Fig. 5, the decreasing discrepancy between Primitive 2 and side distance implies that the internal unit of the agent is gradually specialized to control side distance.

Original: There exists astrong correlation between the discrepancy and the out-of-road rate, which can only be reduced byimproving the ability to keep side distance.
Revised: There exists a strong correlation between the discrepancy and the out-of-road rate, which can only be reduced by improving the ability to keep side distance.

Original: We thus conclude that the pivotal primitives emergeduring training.
Revised: We thus conclude that the pivotal primitives emerge during training.

Original: 4.3 Human-AI Shared Control for Test-time Generalization
Revised: 4.3 Human-AI Shared Control for Test-time Generalization

Original: Using the motor primitives identified above, we can determine the activation value of motor primitivefor steering associated kinematics according to Eq. 8.
Revised: Using the motor primitives identified above, we can determine the activation value of motor primitive for steering associated kinematics according to Eq. 8.

Original: We apply the same process to dissect thepolicy trained from SAC [23] with the same network structure and build a stimulation evoked map forshared control.
Revised: We apply the same process to dissect the policy trained from SAC [23] with the same network structure and build a stimulation-evoked map for shared control.

Original: We then invite human subject to collaborate with the SAC and PPO agents on the testenvironments with different takeover methods.
Revised: We then invite human subject to collaborate with the SAC and PPO agents on the test environments with different takeover methods.

Original: As shown in Tab. 1, the average human involvement,episodic cost, and success rate in the test environments are reported across policies with or withouthuman involvement.
Revised: As shown in Tab. 1, the average human involvement, episodic cost, and success rate in the test environments are reported across policies with or without human involvement.

Original: The human involvement greatly improves the performance and safety whendeploying the policy in unseen environments where delicate maneuvers such as side-passing andyielding are evoked by human, if necessary, to help finish the task.
Revised: The human involvement greatly improves the performance and safety when deploying the policy in unseen environments where delicate maneuvers such as side-passing and yielding are evoked by human, if necessary, to help finish the task.

Original: Different from the original PPO policy which is a MLP with 2 hidden layers, 256 units per layerand “tanh” activation function, we also repeat the same experiment for Deep-PPO agent which has 6hidden layers, and Relu-PPO which has “Relu” as activation function.
Revised: Different from the original PPO policy which is a MLP with 2 hidden layers, 256 units per layer and “tanh” activation function, we also repeat the same experiment for Deep-PPO agent which hashidden layers, and Relu-PPO which has “Relu” as activation function.

Original: The results reported in 1 showour method is invariant to the network architectures and RL algorithms.
Revised: The results reported in 1 show our method is invariant to the network architectures and RL algorithms.

Original: 4.4 Human-AI Shared Control for Task Transfer
Revised: 4.4 Human-AI Shared Control for Task Transfer

Original:  In addition to the improved test-time generalization, the stimulation-evoked map enables RLagent to accomplish new tasks with the help of human.
Revised: [Table] In addition to the improved test-time generalization, the stimulation-evoked map enables RL agent to accomplish new tasks with the help of human.

Original: Recent quadrupedal robots trained by RL exhibit promising ability to walk on bumpy terrain via end-to-end control [35].
Revised: Recent quadrupedal robots trained by RL exhibit promising ability to walk on bumpy terrain via end-to-end control [36].

Original: However,the agent only takes proprioceptive states as input without the observation of surroundings,[Table] and thus it can not avoid obstacles by its own.
Revised: However, the agent only takes proprioceptive states as input without the observation of surroundings, and thus it can not avoid obstacles by its own.

Original: Asshown in Fig.
Revised: As shown in Fig.

Original: 5, we train the agent only to walkon uneven terrain but test it in a new environmentfull of obstacles.
Revised: 4, we train the agent only to walk on uneven terrain but test it in a new environment full of obstacles.

Original: By applying our method, motorprimitives related to yaw rate and speed control canbe found and serve as the interface for human tosteer the “insensible” quadrupedal robot to avoidobstacles.
Revised: By applying our method, motor primitives related to yaw rate and speed control can be found and serve as the interface for human to steer the “insensible” quadrupedal robot to avoid obstacles.

Original: This forms the “State w/oH ” method.
Revised: This forms the “State w/oH ” method.

Original: Wealso include “State+Vision” method [69], which trains a visuomotor policy for legged robot directly inthe test environment, enabling it to sidestep obstacles.
Revised: We also include “State+Vision” method [76], which trains a visuomotor policy for legged robot directly in the test environment, enabling it to sidestep obstacles.

Original: As shown in Table 2, with human involvement, Policy Dissection successfully transforms the “insensible” policy that is not fit to this task and greatlyimproves the performance, achieving comparable performance to the agent trained directly for thistask.
Revised: As shown in Table 2, with human involvement, Policy Dissection successfully transforms the “insensible” policy that is not fit to this task and greatly improves the performance, achieving comparable performance to the agent specially designed for this task. 4.5 Qualitative Result: Parkour Demo As shown in the Fig.

Original: 4.5 Experiment on the Control Precision
Revised: 4, the training objective of the bipedal Cassie robot is to move forward. After training, Policy Dissection can discover several skills in the learned representation, as shown in Fig. Furthermore, these skills can be triggered and controlled by human subject for solving the challenging parkour environment. See the demo video for detail.

Original: [Figure]
Revised: [Figure]

Original: We conduct an experiment to evaluate theprecision of the control  enabled by Policy Dissection.
Revised: We conduct an experiment to evaluate the control precision enabled by our method.

Original: Wefirst train a recent goal-conditionedquadrupedal Animal-C robot in IsaacGym withthe script provided in https://github.com/leggedrobotics/legged_gym .
Revised: We train a goal-conditioned policy for ANYmal-C robot in IsaacGym [57].

Original: The trained robotcan track a heading command and thus serves as thegoal-condition baseline.
Revised: The trained robot taking raw rate as input can track a target yaw with a PID controller, and thus serves as the baseline.

Original: For a fair comparison, we directly dissect this policy and identify the yaw velocityprimitive.
Revised: For a fair comparison, we directly dissect this policy and identify the yaw primitive.

Original: Recall that a positive (negative) activationwill increase (decrease) corresponding kinematic attributes.

Original: Therefore, a PID controller that takes theheading error as input can be used to automaticallydetermine the output of this primitive.
Revised: Likewise, a PID controller that takes the yaw error as input is used to automatically determine the output v j for yaw primitive, enabling tracking target yaw.

Original: Note that forthe primitive activation control the explicit headingcommand is set to 0 in the whole experiment.
Revised: Note that for the primitive activation control the explicit heading command in input is set to 0.

Original: Heading tracking results in Fig.
Revised: Heading tracking results in Fig.

Original: 7 show that the controlachieved by primitive activation can have comparable precision to the goal-conditioned control.
Revised: 7 shows that the control achieved by primitive activation has comparable precision to the goal-conditioned control (see demo video for more detail).

Original:  Conclusion 
Revised:  Conclusion and Discussion

Original: Inspired by the neuroscience approach to study the motor cortex in primates, we present a simpleyet effective method called Policy Dissection to align internal neural representations and kinematicattributes of a given trained policy.
Revised: Inspired by the neuroscience approach to study the motor cortex in primates, we present a simple yet effective method called Policy Dissection to align internal neural representations and kinematic attributes of a pretrained neural policy.

Original: The resulting stimulation-evoked map allows human to steerthe agent and evoke its certain behaviors.
Revised: The resulting stimulation-evoked map allows human to steer the agent and evoke its certain behaviors.

Original: Policy Dissection implements the human-AI sharedcontrol systems on top of the models trained from standard RL algorithms without  re-training ormodification.
Revised: Policy Dissection enables human-AI shared control systems on top of the models trained from standard RL algorithms, without any re-training or modification.

Original: The human-AI shared control systemis quantitatively evaluated in autonomous drivingand locomotion tasks, and the result suggests the  improvement on test-time generalizability and abilityto achieve task transfer.
Revised: We quantitatively evaluate the human-AI shared control system in autonomous driving and locomotion tasks, and the result suggests the substantial improvement of the test-time generalization for task transfer.

Original: We further provide intriguing demonstrations on various other environmentsto show the generality of our method and the wider applications of human-AI collaboration.
Revised: We further provide interactive demonstrations in various environments to show the generality of our method and its application for human-AI collaboration.

Original: Limitations and societal impact .
Revised: Limitations and societal impact .

Original: Our study finds that a small portion of units are not aligned withany motor primitive.

Original: The agent can still accomplish the task even if we set them as zero.

Original: Moreinvestigation is needed for understanding the role of all units and making the neural policy transparent.

Original: Another limitation of Policy Dissection is that we only evaluate this method in continuous controltasks with  state vector as input.
Revised: One limitation of Policy Dissection is that we only evaluate this method in continuous control tasks with the state vector as input.

Original: A wider range of tasks and neural network architecture such asdiscrete tasks and CNNs can be adopted in future research.
Revised: A wider range of tasks and neural network architecture such as discrete control tasks and CNNs can be adopted in future research.

Original: One potential negative social impact isthat the safety issue remains  when deploying the shared control system in real world application.
Revised: One potential negative social impact is that the safety issue remains to be solved when deploying the shared control system in real world application.

Original: As shown in the autonomous driving experiment, our method derived from empirical study has notheoretical guarantee on the safety of human subject.
Revised: As shown in the autonomous driving experiment, our method derived from empirical study has no theoretical guarantee on the safety of human subject.

Original: The unsuccessful collaboration or conflictbetween human and AI may lead to potential dangers.
Revised: The unsuccessful collaboration or conflict between human and AI may lead to potential risks.

Original: Checklist
Revised: A Code and Demo Video.

Original: For all authors... (a) Do the main claims made in the abstract andintroduction accurately reflect the paper’s contributions and scope?
Revised: In the project webpage: https://metadriverse.github.io/policydissect , we provide demonstrative videos showing the Policy Dissection for human-AI shared control and the source code including all trained controllers.

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: B Workflow of Policy Dissection

Original: [Yes] See Section 5.
Revised: After identifying the motion generation block, motor primitives , we can pick one or more of them to activate simultaneously and evoke a certain behavior. Therefore, a group of units are aligned with a certain behavior and stimulation-evoked map can be built.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: The performance of shared controls system is largely determined by the effective coordination between human subject and the AI.

Original: [Yes] See
Revised: Some of our experiment participants report that collaboration with quadrupedal AI is more difficult than with self-driving AI. The possible reason is that human subjects are more familiar with driving vehicles than controlling legged robots. For the shared driving task, human subjects have better global planning abilities about which lane to choose and when to stop. However, for controlling the legged robots on bumpy terrain and avoiding obstacles, some of them didn’t consider how the complex terrain would influence the pose of legged robots and thus had a hard time interacting with the quadrupedal AI.

Original: Section 5, Limitations
Revised: For example, when the robot is moving down from a small slope and is pitching forward, stopping command should be issued cautiously. Otherwise, the agent may roll or flip forward. Therefore, it would be better if human subjects have prior knowledge about the task, so that they can collaborate better with the AI trained for this task. How to implement effective human-AI teaming in general is an important but less explored direction.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: Besides the results shown in the main paper, we provide more qualitative results of human-AI shared control on policies trained in MetaDrive, Pybullet-A1, Ant, Walker and BipedalWalker environments.

Original: them? [Yes]
Revised: As shown in Fig.

Original: If you are including theoretical results...
Revised: 8, in the top-down view we visualize the trajectories of the PPO agent with or without human shared control on the test environment of driving task. Since the PPO agent is trained in the mild traffic density without obstacles, it is incompatible to brake and side-pass. In contrast, with human in the control loop, the lane changing and braking can be deliberately evoked by human, helping the agent solve the near-accidental cases.

Original: (a) Did you state the full set of assumptions of all theoretical results? [N/A](b)
Revised: We provide a summary on how to identify and use motor primitives to evoke behaviors for the trained agents in different environments. [Figure] MetaDrive: Brake.

Original: Did you include complete proofs of all theoretical results?
Revised: By activating the motor primitive positively related to speed with a negative value, the agent can gradually decrease the speed and still stay in the same lane. The brake can be evoked to avoid collision to obstacles cut- vehicles which never MetaDrive: Lane Changing. Since the agent can observe the distance to left and right side, identifying motor primitives related to sidewalk and yellow solid line can make the distance to side controllable.

Original: [N/A]3.
Revised: We use “tanh” as activation function which is symmetric. Therefore identifying only one motor primitive related to one side is enough to evoke lane changing behavior, since, for example, increasing the distance to left side equals decreasing the distance to right side. Lane changing is useful for sidestepping obstacles or moving to another lane with lower traffic density. Pybullet-A1:

Revised: Turning Left/Right.

Original: If you ran experiments...
Revised: As shown in Fig. 9, the moving direction of legged robot can be controlled by activating the neurons related to yaw or yaw rate. Similarly, negative and positive stimulation makes the agent turn towards opposite direction. Pybullet-A1: Stopping.

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: We can also evoke deceleration or stopping behaviors on the legged robot by controlling the activation of speed-related units. Since a huge deceleration may make the agent roll forward, we also add a pitch control to suppress the possible front flipping. Gym-Ant: Stopping. . The ant will stop if motor primitive associated with x-axis speed is activated.

Original: mental results (either in the supplemental material or as a URL)?
Revised: Gym-Ant: Moving Up/Down . The ant can move up/down if we activate two primitives respectively related to: 1) Speed of y-axis and 2) heading direction. Similar to evoking legged robot turning behavior, opposite activation values make the agent move in opposite direction. Gym-Ant: Spinning .

Original: [Yes] See supplemen-
Revised: The ant can spin if we activate unis related to yaw rate. Gym-Walker: Deceleration . The agent trained in Walker stops because we stimulate the motor primitives positively associated with velocity with a negative stimulation.

Revised: As a result, the low-level action sequence for deceleration is executed.

Original: tal material
Revised: Gym-Walker: Hopping . We manage to find the motor primitives associated with torque force. We then disable the movement of the knee in red leg by stimulating the motor primitive with negative value. As a result, the agent hops with only the yellow leg.

Revised: Gym-BidepalWalker: Jumping .

Revised: For the agent trained in BipedalWalker, jumping is achieved by stimulating motor primitives related to V z , where Z-axis is upward.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: Gym-BidepalWalker: Front Flipping . We train the BidepdalWalker walking with larger torque, and thus it can jump higher if we activate units associated with V z . BipedalWalker can conduct front flipping, which is the combination of jumping and pitching. This is resulted from combining another motor primitive related to pitch into control.

Original: how they were chosen)? [Yes] Hyperparameters are in appendix and experimental settings are
Revised: Therefore, the walker jumps with increasing angular velocity, performing front flipping.

Original: in Section
Revised: Gym-BidepalWalker: Standing Up from Split .

Original: (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: In addition, a common failure mode of agent trained in BipedalWalker is that the agent performs split and can not stand up when both legs touch

Original: [Yes]
Revised: [Figure]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g.,
Revised: the ground. We identify the motor primitives associated with all the motor torques.

Original: type of GPUs, internal cluster, or cloud provider)?
Revised: Stimulating these motor primitives , the agent manages to stand up and continue moving forward. IsaacGym-Cassie: Crouching and Tiptoe . The crouching and tiptoe behaviors are related to the z-height of the main robot body, so that we can identify z-axis related units, and activate them to change the z-height. IsaacGym-Cassie: Backflip .

Original: [Yes] See Section 4. Experimental
Revised: This behavior can be described by increasing 1 .

Original: [Table] Setting
Revised: height 2 . pitch and 3 .

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: knee force. Therefore, we activate three corresponding motor primitives selected by Eq. 4 with output calculated by Eq.

Original: [Table]
Revised: 8 to make the robot back flip.

Original: (a) If your work uses existing assets, did you cite the creators? [N/A] (b) Did you mention the license of the assets?
Revised: Note that we also manually determine the termination time t 1 for this action, preventing it from continuously back flipping

Original: [N/A]
Revised: IsaacGym-Cassie: Jumping .

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: The jumping is represented by increasing 1 . knee torque force 2 . forward speed/moving distance.

Original: [N/A]
Revised: Therefore, activating corresponding motor primitives evoke jumping.

Original: [Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table] using/curating?
Revised: Similar as the back-flip, the activation period T = t 1 − t 0 is predetermined, so that it only jump once after pressing the key for jumping.

Original: [N/A]
Revised: IsaacGym-Cassie: Redirection . The redirection can be achieved in the same way as Gym-Ant, activating y-axis movement related units or heading related units. The forward movement can be evoked by activating x-axis related motor primitives .

Original: [Table]
Revised: [Figure]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: We provide a visualization results of the control precision experiments, as shown in Fig.

Original: [Table]
Revised: It is obvious that both both goal-conditioned control methods can drive ANYmal-C to track the yaw command smoothly and stably. The main difference is that the goal-conditioned control enabled by Policy Dissection has a slightly longer response time when changing command, compared to explicit goal-conditioned control.

Original: information or offensive content?
Revised: The demo video is availabe here .

Original: [N/A] [Table] If you used crowdsourcing or conducted research with human subjects...
Revised: We provide more environment details for MetaDrive and Pybullet-A1 such as the observations, the design of reward functions, and the termination conditions.

Original: [Table]
Revised: F.1 MetaDrive

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: In the driving task, the objective of RL agents is to steer the target vehicles with low-level continuous control actions, namely acceleration, brake, and steering.

Original: [Table]
Revised: Observation.

Original: applicable?
Revised: The observation of RL agents is as follows: • A 240-dimensional vector denoting the Lidar-like point clouds with 50 m maximum detecting distance centering at the target vehicle.

Original: [Yes] [Table] (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: Each entry is in [0 , 1] with Gaussian noise and represents the relative distance of the nearest obstacle in the specified direction.

Original: Board (IRB) approvals, if applicable?
Revised: • A vector containing the data that summarizes the target vehicle’s state such as the steering, heading, velocity, and relative distance to the left and right boundaries. • The navigation information that guides the target vehicle toward the destination. We sparsely spread a set of checkpoints, 50m apart on average, in the route and use the relative positions toward future checkpoints as additional observation to the target vehicle.

Original: [Yes] See Section 4.
Revised: Reward and Cost Scheme. The reward function is composed of four parts as follows: [Equation] The displacement reward R disp =

Revised: d t − d t − 1 , wherein the d t and d t − 1 denotes the longitudinal movement of the target vehicle in Frenet coordinates of current lane between two consecutive time steps, provides dense reward to encourage agent to move forward.

Revised: The speed reward R speed = v t /v max incentives agent to drive fast.

Revised: v t and v max denote the current velocity and the maximum velocity ( 80 km/h ), respectively.

Revised: We also define a sparse terminal reward R term , which is non-zero only at the last time step.

Revised: At that step, we set R disp =

Original: {+R speed = 
+}
Revised: {+R speed = 
+}

Revised: and assign R term according to the terminal state.

Revised: R term is set to +10 if the vehicle reaches the destination, − 5 for crashing others or violating the traffic rule.

Revised: We set c 1 = 1 and c 2 = 0 .

Revised: For measuring the safety, collision to vehicles, obstacles, sidewalk raises a cost +1 at each time step.

Revised: The sum of cost generated in one episode is episode cost, a metric like episode reward, but reflecting safety instead.

Revised: Termination Conditions and Evaluation Metrics.

Revised: Since we attempt to benchmark the safety of shared-control system, collision to vehicles and obstacles will not terminate the episode.

Revised: The episode will be terminated only when: 1) the agent drive out of the drivable area and 2) the agent arrives the destination.

Revised: For each trained agent, we evaluate it in 20 held-out test environment and define the ratio of episodes where the agent arrives at the destination as the success rate .

Revised: The definition is the same for out of road .

Revised: Also, the average episode reward and cost on 20 test environment produce two metrics: Episodic Reward and Episodic Cost .

Revised: Since each agent are trained across 5 random seeds, this evaluation process will be executed for 5 agent which has same training setting but different random seeds.

Revised: We report the average and std on the 4 metrics mentioned above.

Revised: F.2 Pybullet-A

Revised: In the quadrupedal locomotion task, the RL training objective is to move forward on bumpy terrain as fast as possible.

Original: Experimental Setting
Revised: We use two environments: the one without obstacles as training environment, and the other one with obstacles as test environment. The environment construction, reward definition, tasks are the same as [76]. Its environment has open-source code at: https://github.com/Mehooz/ vision4leg . In this work, we only slightly modify the observation as follows:

Revised: • IMU recording Yaw, Yaw rate, Pitch and Roll • Angle of 12 joints • Torque applied to 12 joints

Revised: Containing historical output of these sensors over the last 3 steps, the proprioceptive observation is in 84 dimension.

Revised: For LocoTransformer baseline method directly trained on test environment, the input is this state vector and first-view depth images in 64 × 64 over the last 4 steps.

Revised: F.3 IsaacGym

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: We use the same setting reported in [57], and the training code is from https://github.com/ leggedrobotics/legged_gym . G Learning Curves of Trained Agents G.1 MetaDrive [Figure] [Figure]

Revised: G.2 Pybullet-A1 Legged Robots

Revised: [Figure]

Revised: G.3 Gym Environment

Original: [Table]
Revised: [Figure]

Revised: H Hyper-parameters H.1 MetaDrive

Revised: Since SAC is not sensitive to the choice of hyterparameters, the learning rate 1e − 4 is also suitable for

Original: spent on participant compensation?
Revised: Deep-SAC (6 Layer) and Relu SAC. For Deep-PPO and Relu-PPO, learning rate should be slightly decreased due to the change of neural network structure. Also note that the number of hidden units is changed to 128 per layer for Deep PPO with 6 layer. This is for keeping the total number of network variables approximate to the network used by default PPO (around 250,000 variables).

Original: [Yes] See Section 4.
Revised: [Table] [Table] [Table] [Table]

Original: Experimental Setting
Revised: [Table] H.2 Gym Environments and Pybullet-A1 Legged Robots Agents of Gym environments (Walker/Ant/Bidepadel Walker) are trained by SAC and share the same hyperpatrameter setting. Quadrupedal robots are trained by PPO, and we follow the same codebase and configuration used in [76].

Original: [Table] 
Revised: [Table] [Table]