Original: Human-AI Shared Control via Frequency-based Policy Dissection
Revised: Human-AI Shared Control via Frequency-based Policy Dissection

Original: Abstract
Revised: Abstract

Original: Human-AI shared control allows human to interact and collaborate with AI to accomplish control tasks in complex environments.
Revised: Human-AI shared control allows human to interact and collaborate with AI to accomplish control tasks in complex environments.

Original: Previous Reinforcement Learning(RL) methods attempt the goal-conditioned design to achieve human-controllablepolicies at the cost of redesigning the reward function and training paradigm.
Revised: Previous Reinforcement Learning(RL) methods attempted a goal-conditioned design to achieve human-controllablepolicies at the cost of redesigning the reward function and training paradigm.

Original: Inspired by the neuroscience approach to investigate the motor cortex in primates, wedevelop a simple yet effective frequency-based approach called Policy Dissectionto align the intermediate representation of the learned neural controller with thekinematic attributes of the agent behavior.
Revised: Inspired by the neuroscience approach to investigate the motor cortex in primates, wedevelop a simple yet effective frequency-based approach called Policy Dissectionto align the intermediate representation of the learned neural controller with thekinematic attributes of the agent behavior.

Original: Without modifying the neural controlleror retraining the model, the proposed approach can convert a given RL-trainedpolicy into a human-interactive policy.
Revised: Without modifying the neural controlleror retraining the model, the proposed approach can convert a given RL-trainedpolicy into a human-interactive policy.

Original: We evaluate the proposed approach onthe RL tasks of autonomous driving and locomotion.
Revised: We evaluate the proposed approach onthe RL tasks of autonomous driving and locomotion.

Original: The experiments show thathuman-AI shared control achieved by Policy Dissection in driving task can substantially improve the performance and safety in unseen traffic scenes.
Revised: The experiments show thathuman-AI shared control achieved by Policy Dissection in driving task can substantially improve the performance and safety in unseen traffic scenes.

Original: With human inthe loop, the locomotion robots also exhibit versatile controllable motion skills eventhough they are only trained to move forward.
Revised: With human inthe loop, the locomotion robots also exhibit versatile controllable motion skills eventhough they are only trained to move forward.

Original: Our results suggest the promisingdirection of implementing human-AI shared autonomy through interpreting thelearned representation of the autonomous agents.
Revised: Our results suggest the promisingdirection of implementing human-AI shared autonomy through interpreting thelearned representation of the autonomous agents.

Original: Code will be made available.
Revised: Code will be made available.

Original:  Introduction
Revised:  Introduction

Original: In recent years, autonomous agents trained from deep Reinforcement Learning (RL) achieve hugesuccess in a range of applications from robot control [61, 50], autonomous driving [7, 29], to thepower system in smart building [40].
Revised: In recent years, autonomous agents trained from deep Reinforcement Learning (RL) have achievedhuge success in a range of applications from robot control [63, 51], autonomous driving [7, 29], tothe power system in smart building [41].

Original: Despite the capability to effectively discover feasible policyunder unknown system dynamics in a model-free setting [59], the lack of generalizability [14] inunseen environments hinders the real-world deployment of learning-based neural policy.
Revised: Despite the capability to effectively discover feasible policyunder unknown system dynamics in a model-free setting [61], the lack of generalizability [14] inunseen environments hinders the real-world deployment of learning-based neural policy.

Original: Furthermore,it remains difficult to understand the internal mechanism and the decision making of the neuralnetwork [30, 70], especially when abnormal behaviors happen [10, 69, 45].
Revised: Furthermore,it remains difficult to understand the internal mechanism and the decision making of the neuralnetwork [30, 72], especially when abnormal behaviors happen [10, 71, 46].

Original: The lack of transparencyand controllability greatly limit the application of the end-to-end neural controllers in safety-criticalsystems.
Revised: The lack of transparencyand controllability limits the application of the end-to-end neural controllers in safety-critical systems.

Original: One potential direction to make the neural controller safer and more trustworthy is incorporatinghuman into the control loop, which we refer  as human-AI shared control .
Revised: One potential direction to make the neural controller safer and more trustworthy is incorporatinghuman into the control loop, which we refer to as human-AI shared control .

Original: With human involvement,the performance and the safety of the shared autonomy systems can be substantially improved [48, 15,49], even on new tasks and unseen environments.
Revised: With human involvement,the performance and the safety of the shared autonomy systems can be substantially improved [49, 15,50], even on new tasks and unseen environments.

Original: Human-AI shared control has been implementedin various forms, such as enforcing intervention [28, 46] or providing high-level commands [15] tothe AI models.
Revised: Human-AI shared control has been implementedin various forms, such as enforcing intervention [28, 47] or providing high-level commands [15] tothe AI models.

Original: Previous work also explores goal-conditioned reinforcement learning as a form ofhuman-AI shared control, where a high-level goal is fed as an input to the policy during training[53, 2, 15, 25].
Revised: Previous works also explore goal-conditioned reinforcement learning as a form ofhuman-AI shared control, where a high-level goal is fed as an input to the policy during training[55, 2, 15, 25].

Original: Thus the behavior of the learned agent can be controlled by varying the inputgoal [43, 36, 71].
Revised: Thus the behavior of the learned agent can be controlled by varying the inputgoal [44, 36, 73].

Original: However, training goal-conditioned policy requires additional modification of the
Revised: However, training goal-conditioned policy requires additional modification of the

Original: [Figure]
Revised: [Figure]

Original: neural controller, algorithm, and the reward function, while the human control only happens at thegoal level, still lacking interpretability and control of fine-grained movement.
Revised: neural controller, algorithm, and the reward function, while the human control only happens at thegoal level, still lacking interpretability and control of fine-grained movement.

Original: In this work we develop a minimalist approach called Policy Dissection , to enable human-AI sharedcontrol on autonomous agents trained on a wide range of tasks.
Revised: In this work, we develop a minimalist approach called Policy Dissection , to enable human-AI sharedcontrol on autonomous agents trained on a wide range of tasks.

Original: Policy Dissection does not imposeassumption on the agent’s training scheme, and it requires neither retraining the controller’s networknor modifying the environment.
Revised: Policy Dissection does not imposeassumption on the agent’s training scheme, and it requires neither retraining the controller’s networknor modifying the environment.

Original: Policy Dissection achieves human-AI shared control by firstlydissecting the internal representations of the learned policy and aligning them with specific kinematicattributes, and then modulating the activation of the internal representations to elicit the desiredmotor skills.
Revised: Policy Dissection achieves human-AI shared control by firstlydissecting the internal representations of the learned policy and aligning them with specific kinematicattributes, and then modulating the activation of the internal representations to elicit the desiredmotor skills.

Original: Policy Dissection takesthe inspiration from neuroscience studies on motor cortex ofprimates: exerting electrical stimulation on different areas of motor cortex can elicit meaningful bodymovements.
Revised: Policy Dissection takes inspiration from neuroscience studies on motor cortex ofprimates: exerting electrical stimulation on different areas of motor cortex can elicit meaningful bodymovements.

Original: A stimulation-evoked map can be built to reveal the relationship between the evokedbody movement and the motor neurons located in different areas of motor cortex [21, 20, 33].
Revised: A stimulation-evoked map can be built to reveal the relationship between the evokedbody movements and the motor neurons located in different areas of motor cortex [21, 20, 33].

Original: Inorder to replicate the stimulation-evoked map of a neural controller, the proposed Policy Dissectionfollows a frequency analysis and matching strategy, associating the kinematic attribute and the unitactivation.
Revised: Inorder to replicate the stimulation-evoked map of a neural controller, the proposed Policy Dissectionfollows a frequency analysis and matching strategy, associating the kinematic attribute and the unitactivation.

Original: The procedure of building  stimulation-evoked map by Policy Dissection is shown in Fig.
Revised: The procedure of building a stimulation-evoked map by Policy Dissection is shown in Fig.

Original: Concretely, we first roll out the given policy for several episodes.
Revised: Concretely, we first roll out the given policy for several episodes.

Original: After collecting a set of time seriesof the activities of all units, kinematics and dynamics, we identify the units whose activities align withthe kinematic and dynamic attributes in a way that there is the lowest frequency discrepancy betweenthe specific units and the kinematic attributes in the frequency domain.
Revised: After collecting a set of timeseries of the activities of all units, kinematics and dynamics, we identify the units whose activitiesalign with the kinematic and dynamic attributes in a way that there is the lowest frequency discrepancybetween the specific units and the kinematic attributes in the frequency domain.

Original: The stimulation-evoked mapbetween units and attributes is thus built and used to modulate the activation intensity of identifiedunits so that the desired motor skills can be elicited.
Revised: The stimulationevoked map between units and attributes is thus built and used to modulate the activation intensity ofidentified units so that the desired motor skills can be elicited.

Original: Therefore, Policy Dissection constructs aninterpretable control interface on top of the trained agent for human to interact with and evoke certainbehaviors of the agent, thus achieving human-AI shared control.
Revised: Therefore, Policy Dissection constructsan interpretable control interface on top of the trained agent for human to interact with and evokecertain behaviors of the agent, thus achieving human-AI shared control.

Original: We evaluate the proposed method on several RL agents ranging from locomotion robots to autonomousdriving agents in simulation environments.
Revised: We evaluate the proposed method on several RL agents ranging from locomotion robots to autonomousdriving agents in simulation environments.

Original: Experimental results suggest that meaningful motorprimitives emerge in the internal representation.
Revised: Experimental results suggest that meaningful motorprimitives emerge in the internal representation.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, novel behaviors can bedeliberately evoked by activating units related to certain motor primitives, even though they are notthe necessary behaviors to solve the primal task.
Revised: 1, novel behaviors can bedeliberately evoked by activating units related to certain motor primitives, even though they are notthe necessary behaviors to solve the primal task.

Original: In the quantitative evaluation, we use the human-AIshared control system enabled by Policy Dissection to improve the generalization in test-time unseenenvironment and achieve task transfer.
Revised: In the quantitative evaluation, we use the human-AIshared control system enabled by Policy Dissection to improve the generalization in test-time unseenenvironment and achieve task transfer.

Original: In an autonomous driving task, we first train baseline agentsunder mild traffic condition and evaluate the driving policy on new test environments containingunseen near-accidental scenes.
Revised: In an autonomous driving task, we first train baseline agentsunder mild traffic conditions and evaluate the driving policy on new test environments containingunseen near-accidental scenes.

Original: Different to the poor performance of the baseline agent in thesenew environments, the human-AI shared control system achieves superior performance and safetyguarantee, with 95% success rate and almost zero cost.
Revised: Different from the poor performance of the baseline agent in thesenew environments, the human-AI shared control system achieves superior performance and safetyguarantee, with 95% success rate and almost zero cost.

Original: In quadrupedal locomotion task, a robot withonly proprioceptive state input can avoid obstacles with shared control, even though its primal task ismoving forward.
Revised: In quadrupedal locomotion task, a robot withonly proprioceptive state input can avoid obstacles with shared control, even though its primal task is

Original: These results show that the proposed Policy Dissection not only helps understand

Original: [Figure]
Revised: [Figure]

Original:  the  learned representations of the neural controller, but also brings the intriguing potential applicationof human-AI shared control.
Revised: moving forward. These results show that the proposed Policy Dissection not only helps understandthe learned representations of the neural controller, but also brings the intriguing potential applicationof human-AI shared control.

Original:  Related Work
Revised:  Related Work

Original: [Figure]

Original: Human-AI Shared Control .
Revised: Human-AI Shared Control .

Original: Existing methods on human-AI shared control can be roughly dividedinto two categories.
Revised: Existing methods on human-AI shared control can be roughly dividedinto two categories.

Original: The first category is to have human in the control loop to train learningsystems and then execute without human in test time [72].
Revised: The first category is to have human in the control loop to train learningsystems and then execute without human in test time [74].

Original: By incorporating human into the training,previous works successfully improve the performance on visual input control tasks such as Atarigame [1, 51, 63].
Revised: By incorporating human into the training,previous works successfully improve the performance on visual input control tasks such as Atarigame [1, 53, 65].

Original: Robotic control tasks also benefit from human feedback [28, 39, 57, 46, 64].
Revised: Robotic control tasks also benefit from human feedback [28, 40, 59, 47, 66].

Original: Theother category is to have human in both training and test time to accurately accomplish humanassistive tasks.
Revised: Anothercategory is to have human in both training and test time to accurately accomplish human-assistivetasks.

Original: A series of works study this problem based on the Atari Game [48, 52, 8, 31].
Revised: A series of works study this problem based on the Atari Game [49, 54, 8, 31].

Original: Inaddition, the human-AI shared control system is built on autonomous vehicle [19], robotics arm [27]and in multi-agent setting [31].
Revised: In addition,the human-AI shared control system is built on autonomous vehicle [19], robotic arms [27] and inmulti-agent setting [31].

Original: The reported results on various tasks reflect the effectiveness andefficiency of training and coordination with human-assistive AI.
Revised: The reported results on various tasks reflect the effectiveness and efficiencyof training and coordination with human-assistive AI.

Revised: Unlike prior works, our method allows humanto collaborate with AI in test-time without any training-time interventions.

Revised: In experiments, sharedcontrol systems are implemented on several robotic control tasks to show the effectiveness of ourmethod.

Original: Neural Network Interpretability .
Revised: Neural Network Interpretability .

Original: There is a growing interest in  AI community to understand whatis learned inside the neural network.
Revised: There is a growing interest in the AI community to understandwhat is learned inside the neural network.

Original: Previous interpretability methods mostly focus on explainingthe networks trained for visual tasks [73, 55, 11, 74, 38].
Revised: Previous interpretability methods mostly focus on explaining the networks trained for visual tasks [75, 57, 11, 76, 38].

Original: Various empirical studies have found thatmeaningful and interpretable structures emerge in the internal representation when trained for theprimal tasks.
Revised: Various empirical studies have foundthat meaningful and interpretable structures emerge in the internal representation when trained forthe primal tasks.

Original: For example, object detectors emerge in scene classifier [6], and disentangled layoutand attribute controllers emerge from image generation model [66].
Revised: For example, object detectors emerge in scene classifier [6], and disentangled layoutand attribute controllers emerge from image generation model [68].

Original: Manipulating theoutput imageof deep generative models is achieved by editing the disentangled hidden features [5, 56, 65].
Revised: Varying the disentangled hiddenfeatures of the deep generative model can edit the content of the generated images.

Original: On theother hand, visuomotor control tasks can also acquire visual understanding to explain the behaviorchosen by agent with visual input through saliency map [58, 3, 47, 12, 26, 22].
Revised: On theother hand, visuomotor control tasks can also acquire visual understanding to explain the behaviorchosen by agent with visual input through saliency map [60, 3, 48, 12, 26, 22].

Original: Neuroscience Inspired Control .
Revised: Neuroscience Inspired Control .

Original: The complex behaviors shown by recent autonomous agents promptresearchers to develop new analysis methods.
Revised: The complex behaviors shown by recent autonomous agents promptresearchers to develop new analysis methods.

Original: As a source of inspiration, neuroscience providesuseful tools [44, 33], and effective procedures [24, 18, 42] for understanding animal brains which arealso like black-box systems.
Revised: As a source of inspiration, neuroscience providesuseful tools [45, 33], and effective procedures [24, 18, 43] for understanding animal brains which arealso like black-box systems.

Original: Recent works try to develop neural network-based policies from theperspective of neuroscience [17, 4, 68, 13].
Revised: Recent works try to develop neural network-based policies from theperspective of neuroscience [17, 4, 70, 13].

Original: In a closely related work on understanding learned agent,
Revised: In a closely related work on understanding learned agent,

Original: Merel et al.
Revised: Merel et al.

Original: [42] use RSA [34] with CKA [32] similarity index to reveal that behaviors spanningmultiple timescales are encoded in distinct neural layers.
Revised: [43] use RSA [34] with CKA [32] similarity index to reveal that behaviors spanningmultiple timescales are encoded in distinct neural layers.

Original: Apart from providing the neuroscientifictools, many concepts from neuroscience, such as motor primitive also inspire researchers to designgeneralizable controllers [62, 60, 41].
Revised: Apart from providing the neuroscientifictools, many concepts from neuroscience, such as motor primitive also inspire researchers to designgeneralizable controllers [64, 62, 42].

Original:  Policy Dissection Method
Revised:  Policy Dissection Method

Original: Policy Dissection aims at deriving human-controllable policy from a well-trained RL agent.
Revised: Policy Dissection aims at deriving human-controllable policy from a well-trained RL agent.

Original: It hastwo unique features to enable human-AI shared control:
Revised: It hastwo unique features to enable human-AI shared control:

Original: First, it operates on a well-trained RL agentwithout re-training or modifying the original reward/environment and the policy architecture.
Revised: First, it operates on a well-trained RL agent [Figure] without re-training or modifying the original reward/environment and the policy architecture.

Original: Weonly need to roll out the agent for several episodes and log its neural activity and kinematic attributes.
Revised: Weonly need to roll out the agent for several episodes and log its neural activity and kinematic attributes.

Original: Based on the collected records, certain units and kinematic attributes will be aligned.
Revised: Based on the collected records, certain units and kinematic attributes will be aligned.

Original: Thus our [Figure] method is non-intrusive to the primal task and trained model, and applicable to a wide range ofenvironments.
Revised: Thus ourmethod is non-intrusive to the primal task and trained model, and applicable to a wide range ofenvironments.

Original: Second, Policy Dissection can discover generalizable motor skills that even the agentis not trained to perform.
Revised: Second, Policy Dissection can discover generalizable motor skills that even the agentis not trained to perform.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, the Ant robot can be steered to move up/down even itis only trained to move forward as fast as possible in the primal task.
Revised: 1, the Ant robot can be steered to move up/down even itis only trained to move forward as fast as possible in the primal task.

Original: We will introduce the threeessential steps of the Policy Dissection method in detail.
Revised: We will introduce the threeessential steps of the Policy Dissection method in detail.

Original: 3.1 Monitoring Neural Activity and Kinematics
Revised: 3.1 Monitoring Neural Activity and Kinematics

Original: [Figure] Since multi-layer perceptron (MLP) is the most common network architecture used in many RL tasks,we assume the neural controller is a MLP, which has L hidden layers and all layers have the samenumber of I hidden units.
Revised: Since multi-layer perceptron (MLP) is the most common network architecture used in many RL tasks,we assume the neural controller is a MLP, which has L hidden layers and all layers have the samenumber of I hidden units.

Original: We denote a neuron by l, i as z l,i , where l = 1 , ..., L is the index of layerand i = 1 , ..., I is the neuron index in each layer, and its output at time step t is z l,it .
Revised: We denote a neuron by l, i as z l,i , where l = 1 , ..., L is the index of layerand i = 1 , ..., I is the neuron index in each layer, and its output at time step t is z l,it .

Original: The agent’skinematic attributes are represented by s jt , j = 1 , ..., J , where J is the total number of attributes.
Revised: The agent’skinematic attributes are represented by s jt , j = 1 , ..., J , where J is the total number of attributes.

Original: Wedefine the motor primitive as the unit m j ∈ { z } that steers the agent and causes transition of thekinematics.
Revised: We define the motor primitive as the unit m j ∈ { z } that steers the agent and causes transition ofthe kinematics.

Original: We assume that each kinematic attribute s j has a corresponding motor primitive m j todrive the attribute to  desired state, and thus there are also J motor primitives.
Revised: We assume that each kinematic attribute s j has a corresponding motor primitive m jto drive the attribute to a desired state, and thus there are also J motor primitives.

Original: Our goal is to discovermotor primitive m j by associating the units with certain kinematic attribute.
Revised: Our goal is todiscover motor primitive m j by associating the units with certain kinematic attribute.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, we take the Ant robot as an illustrative example.
Revised: 3, we take the Ant robot as an illustrative example.

Original: Policy Dissection starts by rollingout the pretrained agent for several episodes and tracking the activation of all units as well as the thekinematics, such as yaw rate and velocity.
Revised: Policy Dissection starts byrolling out the pretrained agent for several episodes and tracking the activation of all units as well asthe kinematics, such as yaw rate and velocity.

Original: We record the I · L time series { z l,i =
Revised: We record the I · L time series { z l,i =

Original: [ z l,i 1 , z l,i 2 , ... ] } I · Lmeasuring the neural activities and J time series of kinematic attributes { s j
Revised: [ z l,i 1 , z l,i 2 , ... ] } I · Lmeasuring the neural activities and J time series of kinematic attributes { s j

Original: =
Revised: =

Original: [ s j 1 , s j 2 , ... ]
Revised: [ s j 1 , s j 2 , ... ]

Original: } J forfurther analysis.
Revised: } J forfurther analysis.

Original: 3.2 Associating Units with Kinematic Attributes
Revised: 3.2 Associating Units with Kinematic Attributes

Original: Observing the neural activity and kinematics recording in Fig.
Revised: Observing the neural activity and kinematics recording in Fig.

Original: 3, it can be found that several repeatedpatterns of changing kinematics appear in different frequency when executing the policy, whichis reminiscent of animal behavioral research [21, 20, 33].
Revised: 3, we can find that several repeatedpatterns of changing kinematics appear in different frequencies when executing the policy, whichis reminiscent of animal behavioral research [21, 20, 33].

Original: It suggests that the motor skills has itspredominant frequency component [42], controlling fast kinematics and slow kinematics.
Revised: It suggests that the motor skills has itspredominant frequency component [43], controlling fast kinematics and slow kinematics.

Original: Sinceneural activation signals also encode distinct frequency information, this resemblance inspires usto process time series of both neural activity and kinematic attributes in frequency domain, wherethe difference of predominant frequency between two signals can be clearly revealed and the phasediscrepancy of two time series are eliminated.
Revised: Sinceneural activation signals also encode distinct frequency information, this resemblance inspires usto process time series of both neural activity and kinematic attributes in frequency domain, wherethe difference of predominant frequency between two signals can be clearly revealed and the phase discrepancy of two time series are eliminated.

Original: As a consequence, this allows us to associate kinematicattributes with units whose activity best matches the corresponding motor primitives.
Revised: As a consequence, this allows us to associate kinematicattributes with units whose activity best matches the corresponding motor primitives.

Original: In Policy Dissection, all time series are transformed into the frequency domain through Discrete-time
Revised: In Policy Dissection, all time series are transformed into the frequency domain through Discrete-time

Original: Short-time Fourier transform (STFT):
Revised: Short-time Fourier transform (STFT):

Original: [Equation]
Revised: [Equation]

Original: STFT splits the time series x = [ x 1 , x 2 , ... ] into a sequence of temporal window and d ∈ [1 , + ∞ ]denotes the time step of the center of the temporal window.
Revised: STFT splits the time series x = [ x 1 , x 2 , ... ] into a sequence of temporal window and d ∈ [1 , + ∞ ]denotes the time step of the center of the temporal window.

Original: STFT then conducts Fourier transform ineach temporal window.
Revised: STFT then conducts Fourier transform ineach temporal window.

Original: ω ∈ [0 , + ∞ ] denotes the frequency.
Revised: ω ∈ [0 , + ∞ ] denotes the frequency.

Original: To reduce spectral leakage and reservethe frequency information as much as possible, the window function W ( t ) used in Policy Dissectionis blackman window .
Revised: To reduce spectral leakage and reservethe frequency information as much as possible, the window function W ( t ) used in Policy Dissectionis blackman window .

Original: Moreover, the window length is 64 with hop length 16 .
Revised: Moreover, the window length is 64 with hop length 16 .

Original: We apply STFT insteadof naive Fourier transform since it can retain temporal information by breaking the time series intoseveral overlapping chunks of frames.
Revised: We apply STFT insteadof naive Fourier transform since it can retain temporal information by breaking the time series intoseveral overlapping chunks of frames.

Original: This feature is important since in the following frequencymatching we focus on the similarity of resulting spectrograms as well as the temporal correlation oftwo signals.
Revised: This feature is important since in the following frequencymatching we focus on the similarity of resulting spectrograms as well as the temporal correlation oftwo signals.

Original: We use the spectrogram to characterize a measurement:
Revised: We use the spectrogram to characterize a measurement:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: After obtaining the spectrograms for all kinematic attributes and neural activities, frequency discrepancy can be computed for each neuron-kinematics pair ( z l,i , s j ) :
Revised: After obtaining the spectrograms for all kinematic attributes and neural activities, frequency discrepancy can be computed for each neuron-kinematics pair ( z l,i , s j ) :

Original: [Equation]
Revised: [Equation]

Original: F r

Original: [Figure]
Revised: [Equation]

Original: cy

Original: Step

Original: [Figure]
Revised: [Figure]

Original: F r
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: cy

Original: Step
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: M

Original: [Equation]
Revised: [Equation]

Original: For each kinematic attribute, frequency matching isapplied to select the unit with minimal average discrepancy as the motor primitive responsible for thevariation of kinematics:
Revised: For each kinematic attribute, frequency matching isapplied to select the unit with minimal average discrepancy as the motor primitive responsible for thevariation of kinematics:

Original: [Equation]
Revised: [Equation]

Original: where N is the number of collected episodes.
Revised: where N is the number of collected episodes.

Original: Webuild { ( m j , s j ) } , j = 1 , ..., J as the stimulationevoked map, which further enables human to evokecertain behaviors by activating one or more motorprimitives.
Revised: Webuild { ( m j , s j ) } , j = 1 , ..., J as the stimulationevoked map, which further enables human to evokecertain behaviors by activating one or more motorprimitives.

Original: 3.3 Steering Agent via Stimulation-evoked Map
Revised: 3.3 Steering Agent via Stimulation-evoked Map

Original: Human-AI shared can be fulfilled via the stimulation-evoked map.
Revised: Human-AI shared can be fulfilled via the stimulation-evoked map.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, we stimulatecertain motor primitives that are tightly coupled with certain kinematics such as spinning (increasingyaw rate) and deceleration (decreasing linear velocity).
Revised: 2, we stimulatecertain motor primitives that are tightly coupled with certain kinematics such as spinning (increasingyaw rate) and deceleration (decreasing linear velocity).

Original: In this section, we discuss the implementationdetails of the process to elicit the desired behaviors.
Revised: In this section, we discuss the implementationdetails of the process that elicits the desired behaviors.

Original: Identifying Correlation Coefficient .
Revised: Identifying Correlation Coefficient .

Original: The correlation between activation value of motor primitivesand the kinematics attributes needs to be determined.
Revised: The correlation between activation value of motor primitivesand the kinematics attributes needs to be determined.

Original: As shown in Ant robot of Fig.
Revised: As shown in Ant robot of Fig.

Original: 1, activating theneurons controlling y-axis movement with positive value can drive the Ant move left, and a negativevalue makes the Ant move right.
Revised: 1, activating theneurons controlling y-axis movement with positive value can drive the Ant move left, and a negativevalue makes the Ant move right.

Original: We compute the correlation coefficient through spectral phase discrepancy .
Revised: We compute the correlation coefficient through spectral phase discrepancy .

Original: Since motor primitives have their own predominant frequency component responding to specific kinematic attributesresponding in similar frequency, we first find the predominant frequency component:
Revised: we first find the predominant frequency component for each signal:

Original: [Equation]
Revised: [Equation]

Original: At each window of STFT, we can compute the phase discrepancy of the transforms of m j and s j atthe predominant frequency ω ∗ .
Revised: At each window of STFT, we can compute the phase discrepancy of the transforms of m j and s j atthe predominant frequency ω ∗ .

Original: The average phase discrepancy at all temporal windows is:
Revised: The average phase discrepancy at all temporal windows is computedacross N collected episodes:

Original: [Equation]
Revised: [Equation]

Original: where D is the number of temporal windows and Φ is the function retrieving the phase.
Revised: where D is the number of temporal windows and Φ is the function retrieving the phase.

Original: We thennormalize p j to get correlation coefficient ρ between m j and s j as
Revised: We thennormalize p j to get correlation coefficient ρ between m j and s j as

Original: [Equation]
Revised: [Equation]

Original: A positive ρ j indicates we can amplify the kinematic attribute s j by stimulating the motor primitivem j with a positive value.
Revised: A positive ρ j indicates we can amplify the kinematic attribute s j by stimulating the motor primitivem j with a positive value.

Original: Identifying Unit Output .
Revised: Identifying Unit Output .

Original: Suppose we want to amplify the scale of kinematic attribute s j , the outputvalue v j for motor primitive m j , namely the activation of m j , can be determined by is:
Revised: Suppose we want to amplify the scale of kinematic attribute s j , the outputvalue v j for motor primitive m j , namely the activation of m j , can be determined by:

Original: [Equation]
Revised: [Equation]

Original: where k j is a custom constant decided manually for each motor primitive.
Revised: where k j is a custom constant decided manually for each motor primitive.

Revised: Since changing kinematicattribute takes some time, the change of scale will be determined by the length of intervention time.

Revised: For example, if we want to decrease the speed of an autonomous vehicle by activating speed primitive,the longer the activation time, the larger the speed will be decreased.

Revised: In the whole activation period,the output of m j will be overwritten with v j .

Original: Combining Motor Primitives .
Revised: Combining Motor Primitives .

Original: Some behavior requires stimulating one or more motor primitives toevoke.
Revised: Some behavior requires stimulating one or more motor primitives toevoke.

Original: Taking the Ant robot shown in Fig.
Revised: Taking the Ant robot shown in Fig.

Original: 2 as the example, moving towards y-axis can be implemented by two kinematic attributes including increasing V y and heading towards y-axis.
Revised: 2 as the example, moving towards y-axis can be implemented by two kinematic attributes including increasing V y and heading towards y-axis.

Original: Therefore,we activate two corresponding motor primitives selected by Eq.
Revised: Therefore,we activate two corresponding motor primitives selected by Eq.

Original: 4 with value calculated by Eq.
Revised: 4 with value calculated by Eq.

Original: 8 tomake the agent move towards y-axis.
Revised: 8 tomake the agent move towards y-axis.

Revised: The overall policy dissection workflow is in Appendix.

Original:  Experiments
Revised:  Experiments

Original: We first apply policy dissection to an autonomous driving task to study the learning dynamics in
Revised: We first apply policy dissection to an autonomous driving task to study the learning dynamics in

Original: Sec. 4.2.
Revised: Sec. 4.2.

Original: We demonstrate that the emergent pivotal motor primitives are highly correlated withthe success of policy learning.
Revised: We demonstrate that the emergent pivotal motor primitives are highly correlated withthe success of policy learning.

Original: In Sec. 4.3, we show a novel application of the shared autonomy:improving the test-time generalization, where Human-AI shared control can be an alternative toovercoming the generalization gap between training and unseen test-time environment.
Revised: In Sec. 4.3, we show a novel application of the shared autonomy:improving the test-time generalization, where Human-AI shared control can be an alternative toovercoming the generalization gap between training and unseen test-time environment.

Original: The experiment demonstrates  our method finds not only the correlation but causality between unit outputand kinematics.
Revised: The experiment demonstrates that our method finds not only the correlation but causality between unit outputand kinematics.

Original: In Sec. 4.4, we conduct experiments on quadrupedal robot and demonstrate that aquadrupedal robot trained to traverse complex terrain with only proprioceptive state can transfer toaccomplish obstacle avoidance task with a bit human effort, even it does not have any information ofthe surrounding. 
Revised: In Sec. 4.4, we conduct experiments on quadrupedal robot and demonstrate that aquadrupedal robot trained to traverse complex terrain with only proprioceptive state can transfer toaccomplish obstacle avoidance task with a bit human effort, even it does not have any informationof the surrounding. In sec.

Revised: 4.5, we investigate the command tracking accuracy of Policy Dissection,and compare it with the state-of-the-art goal-conditioned controller in IsaacGym [39, 52].

Original: Lastly, more qualitative demonstrations in Sec. 4.5 show that our method can begeneralized to various environments and autonomy tasks.4.
Revised: Lastly,more qualitative demonstrations in appendix show that our method can be generalized to variousenvironments and autonomy tasks.

Original: 1 Experimental Setting
Revised: 4.1 Experimental Setting

Original: The qualitative experiments are conducted on the Ant and Walker in MuJoCo [61] and the BipedalWalker [9] following the general setting.
Revised: The qualitative experiments are conducted on the Ant and Walker in MuJoCo [63] and the BipedalWalker [9] following the general setting.

Original: The quantitative experiments of human-AI shared controlare conducted on MetaDrive and Pybullet-AI environments introduced as follows:
Revised: The quantitative experiments of human-AI shared controlare conducted on MetaDrive and Pybullet-AI environments introduced as follows:

Original: MetaDrive .
Revised: MetaDrive .

Original: We train agents to accomplish autonomous driving task in 
Revised: We train agents to accomplish autonomous driving task in the safety environment of

Original: the

Original: safety environment

Original: of MetaDrive 
Revised: MetaDrive [37].

Original: In this task, the goal is to maneuver the target vehicle to destination with low-levelcontinuous control actions.
Revised: In this task, the goal is to maneuver the target vehicle to destination with low-levelcontinuous control actions.

Original: An episode terminates only when the vehicle arrives  the destination  drives out of road or collides with the sidewalk.
Revised: An episode terminates only when the vehicle arrives at the destination,drives out of road, or collides with the sidewalk.

Original: The observation

Original: of

Original:  autonomous vehicle consists of map, navigation lidar
Revised: The observation of autonomous vehicle consists ofmap, navigation, lidar

Original: and state information such as speed and heading
Revised: and state information such as speed and heading.

Original: Pybullet A1 .
Revised: Pybullet-A1 .

Original: The legged robots locomotion experiments are conducted on the Pybullet-UnitreeA1 [16] robot.
Revised: The legged robots locomotion experiments are conducted on the Pybullet Unitree-  A1 [

Original:  We follow the same environment setting used by Locotransformer robot [67], includingterrain shape, obstacle distributions, sensors, reward definition, and termination condition.
Revised: ] robot We follow the same environment setting used by Locotransformer robot [69], including terrain shape, obstacle distributions, sensors, reward definition, and termination condition.

Original: Two agentsare trained.
Revised: Two agentsare trained.

Original: One “insensible” agent with only proprioceptive state is trained to walk on uneven terrain,while the other agent with Locotransformer is directly trained to avoid obstacle.
Revised: One “insensible” agent with only proprioceptive state is trained to walk on uneven terrain,while the other agent with Locotransformer is directly trained to avoid obstacles.

Original: The shared controlsystem will be built on the “insensible” agent, where human subjects help the agent sidestep obstacles. 
Revised: The shared controlsystem will be built on the “insensible” agent, where human subjects help the agent sidestep obstacles. T r a i n i ng E n v i r on m e n t T es t E n v i r on m e n t MetaDrive SafetyEnv

Revised: Pybullet-A

Revised: [Figure]

Original: Training Set and Test Set .
Revised: Training Set and Test Set .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5, for both autonomous driving and robot locomotiontasks, we prepare training set and held-out test set.
Revised: 5, for both autonomous driving and robot locomotiontasks, we prepare training set and held-out test set.

Original: For driving task, the test set is used to benchmarkthe test-time generalizability of human-AI shared control system and AI-only control system.
Revised: For driving task, the test set is used to benchmarkthe test-time generalizability of human-AI shared control system and AI-only control system.

Original: Wetrain autonomous driving policies in 50 different environments where the traffic condition is mild andall traffic vehicles and  target vehicle can drive smoothly towards their destinations.
Revised: Wetrain autonomous driving policies in 50 different environments where the traffic condition is mild(averagely 2 cars per map) and all traffic vehicles and the target vehicle can drive smoothly towardstheir destinations.

Original: No obstacles arein these environments.
Revised: No obstacles are in these environments.

Original: In testing time, the trained agents will be evaluated in the other 20 unseen T r a i n i ng E n v i r on m e n t T es t E n v i r on m e n t [Figure]
Revised: In testing time, the trained agents will beevaluated in the other 20 unseen maps with higher traffic density (averagely 6 cars per map).

Original: maps with higher traffic density.

Original: Besides, obstacles like traffic cones and breakdown vehicles willscatter randomly on the road.
Revised: Besides,obstacles like traffic cones and breakdown vehicles will scatter randomly on the road.

Original: In this case, the target vehicle needs to be maneuvered delicately.
Revised: In this case,the target vehicle needs to be maneuvered delicately.

Original: Wecompare the performance and safety of the trained policy in two scenarios: runs in the test set solely,and runs with the human-AI shared control system built from Policy Dissection.
Revised: We compare the performance and safety of thetrained policy in two scenarios: runs in the test set solely, and runs with the human-AI shared controlsystem built from Policy Dissection.

Original: The primal task for  legged robot is to move toward one direction as fast as possible.
Revised: The primal task for the legged robot is to move toward one direction as fast as possible.

Original: We designthe test environment to be a forest-like environment where tree obstacles are randomly scattered.
Revised: We designthe test environment to be a forest-like environment where tree obstacles are randomly scattered.

Original: Therefore, though the overall goal is still to move forward, the agent needs extra effort to sidestepobstacles in the test environment.
Revised: Therefore, though the overall goal is still to move forward, the agent needs extra effort to sidestepobstacles in the test environment.

Original: Evaluation Metrics .
Revised: Evaluation Metrics .

Original: For driving task, we evaluate agents on all the 20 test environments, and reportthe ratio of episodes where the agent arrives at the destination as the success rate .
Revised: For driving task, we evaluate agents on all the 20 test environments, and reportthe ratio of episodes where the agent arrives at the destination as the success rate .

Original: Also, Out of Roadrate can be calculated if the termination is caused by driving out of road.
Revised: Also, Out of Roadrate can be calculated if the termination is caused by driving out of road.

Original: For benchmarking the safetyof agents, the collision to vehicles, obstacles, sidewalk and buildings raises a cost +1 .
Revised: For benchmarking the safetyof agents, the collision to vehicles, obstacles, sidewalk and buildings raises a cost +1 .

Original: In addition,human attention cost is a huge concern of human-AI shared control system.
Revised: In addition,human attention cost is a huge concern of human-AI shared control system.

Original: We quantify this usinghuman involvement , which is measured by the number of environment steps when the human subjectpresses keys for activating units, or controls throttle, brake, and steering wheel.
Revised: We quantify this usinghuman involvement , which is measured by the number of environment steps when the human subjectpresses keys for activating units, or controls throttle, brake, and steering wheel.

Original: We average the sumof reward, cost and human involvement on 20 environments as episodic return , episodic cost andhuman involvement .
Revised: We average the sumof reward, cost and human involvement on 20 environments as episodic return , episodic cost andhuman involvement .

Original: The  evaluation processabove will be repeated 5 times for each agent, and wereport the average value and std for all metrics.
Revised: The above evaluation process will be repeated 5 times for each agent, and wereport the average value and std for all metrics.

Original: For the locomotion task, we use moving distanceto measure the performance.
Revised: For the locomotion task, we use moving distanceto measure the performance.

Original: Moving distance is the displacement of robot on specific direction, i.e.x-axis.
Revised: Moving distance is the displacement of robot on specific direction, i.e.x-axis.

Original: The evaluation is also repeated for 5 times on the exclusive test environment.
Revised: The evaluation is also repeated for 5 times on the exclusive test environment.

Revised: Human Interface.

Revised: We use keyboard as interface in all human-AI shared control experiments.

Revised: When aspecific key is pressed, related motor primitives will be activated with output v j constantly accordingto Eq. 8 until another key is pressed, which will trigger new motor primitives.

Revised: Besides, one key mapsto none of the motor primitives which is pressed to recover the default policy and the natural state.

Original: All experiments without human are repeated 5 times with different random seeds.
Revised: All experiments without human are repeated 5 times with different random seeds.

Original: N = 5 episodesare used for applying Policy Dissection.
Revised: N = 5 episodesare used for applying Policy Dissection.

Original: For experiments involving human subject, we repeat eachexperiment with 5 human subjects.
Revised: For experiments involving human subject, we repeat eachexperiment with 5 human subjects.

Original: For driving task, all human subjects have driving license.
Revised: For driving task, all human subjects have driving license.

Original: Humansubject can pause the experiments if any discomfort happens.
Revised: Humansubject can pause the experiments if any discomfort happens.

Original: No human subjects were harmed in theexperiments.
Revised: No human subjects were harmed in theexperiments.

Original: Human subjects have signed the experiment agreement and get compensation. 
Revised: Human subjects have signed the experiment agreement and get compensation. [Table]

Original: 4.2 Policy Dissection for Understanding Learning Dynamics
Revised: 4.2 Policy Dissection for Understanding Learning Dynamics

Original: In this section, we verify the correlation between motor primitives and relevant kinematic attributes.
Revised: In this section, we verify the correlation between motor primitives and relevant kinematic attributes.

Original: We first train PPO [54] agent on the MetaDrive’s training environments and then conduct Policy
Revised: We first train PPO [56] agent on the MetaDrive’s training environments and then conduct Policy

Original: Dissection on the trained policy and find two motor primitives in which Primitive 1 is associatedwith Speed and Primitive 2 associated with Side Distance .
Revised: Dissection on the trained policy and find two motor primitives in which Primitive 1 is associatedwith Speed and Primitive 2 associated with Side Distance .

Original: We then plot the evolution of thefrequency discrepancy between the identified motor primitives and the kinematics alongside with themacroscopic measure of the learning progress in Fig. 
Revised: We then plot the evolution of thefrequency discrepancy between the identified motor primitives and the kinematics alongside withthe macroscopic measure of the learning progress in Fig. Note that the frequency discrepanciesd =

Revised: [ d 1 , ..., d T ] is normalized to [0 , 1] by ˜ d

Revised: = d − min( d ) max( d ) − min( d ) .

Original: In the upper plot in Fig.
Revised: The upper plot in Fig.

Original: 6, the decreasing discrepancy indicates that Primitive 1  to control speedgradually emerge.
Revised: 6 shows Primitive 1 gradually emerges to control speed, indicated by thedecreasing discrepancy.

Original: With strong negative correlation, the success rate rapidly increases when thediscrepancy of speed primitive also rapidly decrease between epoch 20-30.
Revised: With strong negative correlation, the success rate rapidly increases whenthe discrepancy of speed primitive also rapidly decreases between epoch 20-30.

Original: Similarly, in the lowerfigure in Fig. 6, the decreasing discrepancy between Primitive 2 and side distance shows that the [Table]
Revised: Similarly, in thelower plot in Fig. 6, the decreasing discrepancy between Primitive 2 and side distance indicatesthat the internal unit of the agent is gradually specialized to control side distance.

Original: H n requires human to provide steering and throttle in the whole takeover period, while H ∗ only asks to activate motor primitives at the start of takeover.

Original: We also do ablation study on the activation function and number of hidden layers of

Original: MLP.

Original: The results show our method is general and robust to the policy architecture and RL methods.

Original: internal unit of the agent is gradually specialized for controlling side distance.

Original: There exists strongcorrelation between the discrepancy and the out-of-road rate, which can only be reduced by improvingthe ability to keep side distance.
Revised: There exists astrong correlation between the discrepancy and the out-of-road rate, which can only be reduced byimproving the ability to keep side distance.

Original: We thus conclude that the pivotal primitives emerge during training.
Revised: We thus conclude that the pivotal primitives emergeduring training.

Original: 4.3 Human-AI Shared Control for Test-time Generalization
Revised: 4.3 Human-AI Shared Control for Test-time Generalization

Original: Using the motor primitives identified above, we can determine the activation value of motor primitivefor steering associated kinematics according to Eq. 8.
Revised: Using the motor primitives identified above, we can determine the activation value of motor primitivefor steering associated kinematics according to Eq. 8.

Original: We apply the same process to dissect thepolicy trained from SAC [23] with  same network structure and build a stimulation evoked map forshared control.
Revised: We apply the same process to dissect thepolicy trained from SAC [23] with the same network structure and build a stimulation evoked map forshared control.

Original: We then invite human subject to collaborate with the SAC and PPO agents on the testenvironments with different takeover methods.
Revised: We then invite human subject to collaborate with the SAC and PPO agents on the testenvironments with different takeover methods.

Original: As shown in Tab. 1, the average human involvement,episodic cost, and success rate on the test environments are reported across policies with or withouthuman involvement.
Revised: As shown in Tab. 1, the average human involvement,episodic cost, and success rate in the test environments are reported across policies with or withouthuman involvement.

Original: The human involvement greatly improves the performance and safety whendeploying the policy in unseen environments where delicate maneuvers such as side-passing andand yielding are evoked by human, if necessary, to help finish the task.
Revised: The human involvement greatly improves the performance and safety whendeploying the policy in unseen environments where delicate maneuvers such as side-passing andyielding are evoked by human, if necessary, to help finish the task.

Original: Compared to naive takeoverbaseline indicating by H n where human provide control signals in the whole takeover period, we findthat the shared control mechanism enabled by our method only requires a small amount of humanbudget, since human only needs to activate the neuron at the start of takeover period.

Original: Different from the original PPO policy which is a MLP with 2 hidden layers, 256 units per layerand “tanh” activation function, we also repeat the same experiment for Deep-PPO agent which has 6hidden layers, and Relu-PPO which has Relu as activation function.
Revised: Different from the original PPO policy which is a MLP with 2 hidden layers, 256 units per layerand “tanh” activation function, we also repeat the same experiment for Deep-PPO agent which has 6hidden layers, and Relu-PPO which has “Relu” as activation function.

Original: The results reported in 1 showour method is invariant to the network architectures and RL algorithms.
Revised: The results reported in 1 showour method is invariant to the network architectures and RL algorithms.

Original: 4.4 Human-AI Shared Control for Task Transfer
Revised: 4.4 Human-AI Shared Control for Task Transfer

Original: In addition to the improved test-time generalization, the stimulation-evoked map enables RL agentto accomplish new task with the help of human.
Revised: In addition to the improved test-time generalization, the stimulation-evoked map enables RLagent to accomplish new tasks with the help of human.

Original: Recent quadrupedal robots trained by RL exhibitpromising ability to walk on bumpy terrain via end-to-end control [35].
Revised: Recent quadrupedal robots trained by RL exhibit promising ability to walk on bumpy terrain via end-to-end control [35].

Original: However, the agent only 
Revised: However,the agent only takes proprioceptive states as input without the observation of surroundings,

Original: [Table]
Revised: [Table]

Original: takes proprioceptive states as input without the observation of surroundings, and thus it can not avoidobstacles by its own.
Revised: and thus it can not avoid obstacles by its own.

Original: As shown in Fig.
Revised: Asshown in Fig.

Original: 5, we trainthe agent only to walk on uneven terrain but test itin a new environment full of obstacles.
Revised: 5, we train the agent only to walkon uneven terrain but test it in a new environmentfull of obstacles.

Original: By applyingour method, motor primitives related to yaw rate andspeed control can be found and serve as the interface for human to steer the “insensible” quadrupedal[Figure] robot to avoid obstacles.
Revised: By applying our method, motorprimitives related to yaw rate and speed control canbe found and serve as the interface for human tosteer the “insensible” quadrupedal robot to avoidobstacles.

Original: This forms the “State w/oH ” method.
Revised: This forms the “State w/oH ” method.

Original: We also include “State+Vision”method [67], which trains a visuomotor policy for legged robot directly in the test environment,enabling it to sidestep obstacles.
Revised: Wealso include “State+Vision” method [69], which trains a visuomotor policy for legged robot directly inthe test environment, enabling it to sidestep obstacles.

Original: As shown in Table 2, with human involvement, Policy Dissectionsuccessfully transforms the “insensible” policy that is not fit to this task and greatly improves theperformance, achieving comparable performance to the agent trained directly for this task.
Revised: As shown in Table 2, with human involvement, Policy Dissection successfully transforms the “insensible” policy that is not fit to this task and greatlyimproves the performance, achieving comparable performance to the agent trained directly for thistask.

Original: 4.5 Demonstration of Human-AI Shared Control in Diverse Environments
Revised: 4.5 Experiment on the Control Precision

Original: As shown in Fig.
Revised: [Figure]

Original: 7, in the top-down view we visualize the trajectories of thePPO agent with orwithout human shared control on the test environment of driving task. Since the PPO agent is trainedin the mild traffic density without obstacles, it is incompatible to brake and side-pass.
Revised: We conduct an experiment to evaluate the precision of the control enabled by Policy Dissection. We first train a recent goal-conditionedquadrupedal Animal-C robot in IsaacGym withthe script provided in https://github.com/leggedrobotics/legged_gym .

Original: In contrast,with human in the control loop, the lane changing and braking can be deliberately evoked by human,helping the agent solve the near-accidental cases. We provide more qualitative results of human-AI shared control on policies trained in MetaDrive,
Revised: The trained robotcan track a heading command and thus serves as thegoal-condition baseline. For a fair comparison, we directly dissect this policy and identify the yaw velocityprimitive. Recall that a positive (negative) activationwill increase (decrease) corresponding kinematic attributes.

Original: Pybullet-A1, Ant, Walker and BipedalWalker environments.
Revised: Therefore, a PID controller that takes theheading error as input can be used to automaticallydetermine the output of this primitive.

Original: Please find in the supplemental materialfor details about identifying motor primitives and evoking these behaviors. We also include a demovideo of human-AI shared control in  the supplementary material.
Revised: Note that forthe primitive activation control the explicit headingcommand is set to 0 in the whole experiment. Heading tracking results in Fig. 7 show that the controlachieved by primitive activation can have comparable precision to the goal-conditioned control.

Original:  Conclusion
Revised:  Conclusion

Original: Inspired by the neuroscience approach to study the motor cortex in primates, we present a simpleyet effective method called Policy Dissection to align internal neural representations and kinematicattributes of a given trained policy.
Revised: Inspired by the neuroscience approach to study the motor cortex in primates, we present a simpleyet effective method called Policy Dissection to align internal neural representations and kinematicattributes of a given trained policy.

Original: The resulting stimulation-evoked map allows human to steerthe agent and evoke its certain behaviors.
Revised: The resulting stimulation-evoked map allows human to steerthe agent and evoke its certain behaviors.

Original: Policy Dissection implements the human-AI shared [Figure] control systems on top of the models trained from standard RL algorithms without re-training ormodification.
Revised: Policy Dissection implements the human-AI sharedcontrol systems on top of the models trained from standard RL algorithms without re-training ormodification.

Original: The human -AI shared control system is quantitatively evaluated in autonomous drivingand locomotion tasks, and the result suggests the improvement on test-time generalizability and ability to achieve task transfer.
Revised: The human-AI shared control system is quantitatively evaluated in autonomous drivingand locomotion tasks, and the result suggests the improvement on test-time generalizability and abilityto achieve task transfer.

Original: We further provide intriguing demonstrations on various other environments to show the generality of our method and the wider applications of human AI collaboration.
Revised: We further provide intriguing demonstrations on various other environmentsto show the generality of our method and the wider applications of human-AI collaboration.

Original: Limitations and societal impact .
Revised: Limitations and societal impact .

Original: Our study finds that a small portion of units are not aligned withany motor primitive.
Revised: Our study finds that a small portion of units are not aligned withany motor primitive.

Original: The agent can still accomplish the task even if we set them as zero.
Revised: The agent can still accomplish the task even if we set them as zero.

Original: Moreinvestigation is needed for understanding the role of all units and making the neural policy transparent.
Revised: Moreinvestigation is needed for understanding the role of all units and making the neural policy transparent.

Original: Another limitation of Policy Dissection is that we only evaluate this method in continuous controltasks with state vector as input.
Revised: Another limitation of Policy Dissection is that we only evaluate this method in continuous controltasks with state vector as input.

Original: Wider range of tasks and neural network architecture such as discretetasks and CNNs can be adopted in future research.
Revised: A wider range of tasks and neural network architecture such asdiscrete tasks and CNNs can be adopted in future research.

Original: One potential negative social impact is that thesafety issue remains when deploying the shared control system in real world application.
Revised: One potential negative social impact isthat the safety issue remains when deploying the shared control system in real world application.

Original: As shownin the autonomous driving experiment, our method derived from empirical study has no theoreticalguarantee on the safety of human subject.
Revised: As shown in the autonomous driving experiment, our method derived from empirical study has notheoretical guarantee on the safety of human subject.

Original: The unsuccessful collaboration or conflict between humanand AI may lead to potential dangers.
Revised: The unsuccessful collaboration or conflictbetween human and AI may lead to potential dangers.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See
Revised: [Yes] See

Original: Section 5, Limitations
Revised: Section 5, Limitations

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A](b)

Original: (b) Did you include complete proofs of all theoretical results?
Revised: Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] See supplemen-
Revised: [Yes] See supplemen-

Original: tal material
Revised: tal material

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] Hyperparameters are in appendix, and experimental settings are
Revised: [Yes] Hyperparameters are in appendix and experimental settings are

Original: in Section
Revised: in Section

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g.,
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g.,

Original: type of GPUs, internal cluster, or cloud provider)?
Revised: type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Revised: Experimental

Revised: [Table]

Revised: Setting

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original:  (b) Did you mention the license of the assets?
Revised: [Table] (a) If your work uses existing assets, did you cite the creators? [N/A] (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original:  (d) Did you discuss whether and how consent was obtained from people whose data you’re 
Revised: [Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original:  (e) Did you discuss whether the data you are using/curating contains personally identifiable 
Revised: [Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Revised: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original:  (a) Did you include the full text of instructions given to participants and screenshots, if 
Revised: [Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]

Original: applicable?
Revised: applicable?

Original: [Yes]
Revised: [Yes]

Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: Experimental Setting
Revised: Experimental Setting

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Revised: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: Experimental Setting
Revised: Experimental Setting

Revised: [Table]