Original: R EWARD - FREE P OLICY L EARNING THROUGH A CTIVE H UMAN I NVOLVEMENT
Revised: R EWARD - FREE P OLICY L EARNING THROUGH A CTIVE H UMAN I NVOLVEMENT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite the success of reinforcement learning (RL) in many control tasks, the behaviors of the learned agents are largely limited by the hand-crafted reward function in the environment, which might not truthfully reflect human intents and preferences.
Revised: Despite the success of reinforcement learning (RL) in many control tasks, the behaviors of the learned agents are largely limited by the hand-crafted reward function in the environment, which might not truthfully reflect human intents and preferences.

Original: This work proposes a reward-free policy learning method called Proxy
Revised: This work proposes a reward-free policy learning method called Proxy

Original: Value Propagation that conveys human intents explicitly to the learning policy through active involvement.
Revised: Value Propagation that conveys human intents explicitly to the learning policy through active involvement.

Original: We adopt an interactive learning setting where human subjects can actively intervene and demonstrate to the agent.
Revised: We adopt an interactive learning setting where human subjects can actively intervene and demonstrate to the agent.

Original: Our key insight is that a latent value function can be learned from active human involvement, which in return guides the learning policy to emulate human behaviors.
Revised: Our key insight is that a latent value function can be learned from active human involvement, which in return guides the learning policy to emulate human behaviors.

Original: The proposed method first relabels and propagates the proxy values of human demonstrations to other states, and then optimizes the policies to comply with the human intents expressed through the proxy value function.
Revised: The proposed method first relabels and propagates the proxy values of human demonstrations to other states, and then optimizes the policies to comply with the human intents expressed through the proxy value function.

Original: The proposed method can be incorporated into many existing RL algorithms with minimum modifications.
Revised: The proposed method can be incorporated into many existing RL algorithms with minimum modifications.

Original: Experiments on various tasks and human control devices demonstrate the generality and efficiency of our method.
Revised: Experiments on various tasks and human control devices demonstrate the generality and efficiency of our method.

Original: Theoretic guarantee on the learning safety is also provided.
Revised: Theoretic guarantee on the learning safety is also provided.

Original: Demo video and code are available in the supplementary material.
Revised: Demo video and code are available in the supplementary material.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) has been successfully applied in many domains, ranging from board game Go (Silver et al., 2016), strategy game StarCraft II (Samvelyan et al., 2019), autonomous driving (Kendall et al., 2019), and even nuclear fusion (Degrave et al., 2022).
Revised: Reinforcement learning (RL) has been successfully applied in many domains, ranging from board game Go (Silver et al., 2016), strategy game StarCraft II (Samvelyan et al., 2019), autonomous driving (Kendall et al., 2019), and even nuclear fusion (Degrave et al., 2022).

Original: Existing RL methods assume the manually designed reward function can fully express human intents and goals, however, the resulting agents might exhibit biased, misguided, or undesired behaviors due to faulty reward functions (Leike et al., 2018; Russell, 2019; Krakovna et al., 2020).
Revised: Existing RL methods assume the manually designed reward function can fully express human intents and goals, however, the resulting agents might exhibit biased, misguided, or undesired behaviors due to faulty reward functions (Leike et al., 2018; Russell, 2019; Krakovna et al., 2020).

Original: Moreover, the poor sample efficiency as well as the safety concern due to the trial-and-error exploration prevent the real-world deployment of RL.
Revised: Moreover, the poor sample efficiency as well as the safety concern due to the trial-and-error exploration prevent the real-world deployment of RL.

Original: Human-in-the-loop policy learning (HL) methods are promising complements to RL and reward engineering, which relies on human subject to oversee the learning process of the robots and autonomous agents.
Revised: Human-in-the-loop policy learning (HL) methods are promising complements to RL and reward engineering, which relies on human subject to oversee the learning process of the robots and autonomous agents.

Original: HL methods can learn complex and safety-aware behaviors that are intractable to be encoded in the handcrafted reward function.
Revised: HL methods can learn complex and safety-aware behaviors that are intractable to be encoded in the handcrafted reward function.

Original: Different forms of human involvement have been studied over the years.
Revised: Different forms of human involvement have been studied over the years.

Original: Human subjects can advise actions upon the requests of the robots (Mandel et al., 2017) or provide preference-based feedback to assess the relative value of the collected trajectories (Wirth et al., 2017; Christiano et al., 2017; Reddy et al., 2018;
Revised: Human subjects can advise actions upon the requests of the robots (Mandel et al., 2017) or provide preference-based feedback to assess the relative value of the collected trajectories (Wirth et al., 2017; Christiano et al., 2017; Reddy et al., 2018;

Original: Warnell et al., 2018; Palan et al., 2019;
Revised: Warnell et al., 2018; Palan et al., 2019;

Original: Guan et al., 2021).
Revised: Guan et al., 2021).

Original: These methods learn from passive human involvement, where the humans do not provide real-time feedback and intervention during data collection, reducing the efficiency of the human-robot systems.
Revised: These methods learn from passive human involvement, where the humans do not provide real-time feedback and intervention during data collection, reducing the efficiency of the human-robot systems.

Original: On the other hand, an increasing body of works focuses on active human involvement, where human subjects are authorized to actively intervene and provide demonstrations during the execution of robots (Kelly et al., 2019; Spencer et al., 2020;
Revised: On the other hand, an increasing body of works focuses on active human involvement, where human subjects are authorized to actively intervene and provide demonstrations during the execution of robots (Kelly et al., 2019; Spencer et al., 2020;

Original: Mandlekar et al., 2020; Li et al., 2022b).
Revised: Mandlekar et al., 2020; Li et al., 2022b).

Original: With online corrective demonstrations from human subjects, the safety and learning efficiency of the system can be greatly improved.
Revised: With online corrective demonstrations from human subjects, the safety and learning efficiency of the system can be greatly improved.

Original: In this work, we explore transforming a standard RL task into a reward-free setting where the agent can learn from active human involvement.
Revised: In this work, we explore transforming a standard RL task into a reward-free setting where the agent can learn from active human involvement.

Original: Our intuition is that the agent should either replicate the behaviors of human subjects in the circumstances where humans are once involved and provide demonstrations, or should apply the actions that can move toward those human-involved states where high-quality demonstrations are available.
Revised: Our intuition is that the agent should either replicate the behaviors of human subjects in the circumstances where humans are once involved and provide demonstrations, or should apply the actions that can move toward those human-involved states where high-quality demonstrations are available.

Original: We find that this intuition can be instantiated into many existing RL algorithms, by casting active human involvement as human subjects depicting the landscape of a latent value function .
Revised: We find that this intuition can be instantiated into many existing RL algorithms, by casting active human involvement as human subjects depicting the landscape of a latent value function .

Original: We design a method called Proxy Value Propagation (PVP) that can be incorporated into common value-based RL methods.
Revised: We design a method called Proxy Value Propagation (PVP) that can be incorporated into common value-based RL methods.

Original: PVP assigns high Q values to human actions and low values to agent actions that are intervened by human subjects.
Revised: PVP assigns high Q values to human actions and low values to agent actions that are intervened by human subjects.

Original: RL policy thus tends to replicate human actions due to the value-maximizing nature.
Revised: RL policy thus tends to replicate human actions due to the value-maximizing nature.

Original: The proxy values will be propagated to prior states through TD learning, informing that human demonstrations are available in these human-involved states.
Revised: The proxy values will be propagated to prior states through TD learning, informing that human demonstrations are available in these human-involved states.

Original: Experiments show that PVP can be successfully applied to both continuous-action and discrete-action tasks, and achieve higher learning efficiency compared to baselines in solving grid-world navigation, Atari video game, and driving tasks.
Revised: Experiments show that PVP can be successfully applied to both continuous-action and discrete-action tasks, and achieve higher learning efficiency compared to baselines in solving grid-world navigation, Atari video game, and driving tasks.

Original: It is also compatible with various forms of human input devices, including gamepad, driving wheel, and keyboard.
Revised: It is also compatible with various forms of human input devices, including gamepad, driving wheel, and keyboard.

Original: We summarize our main contributions as follows:
Revised: We summarize our main contributions as follows:

Original: 1) We show that active human involvement is an effective supervision to train the agent and theoretically prove the safety guarantee in training and testing time under this setting.
Revised: 1) We show that active human involvement is an effective supervision to train the agent and theoretically prove the safety guarantee in training and testing time under this setting.

Original: We propose a simple yet effective method Proxy Value Propagation that can be easily integrated in existing RL algorithms to learn from active human involvement.
Revised: We propose a simple yet effective method Proxy Value Propagation that can be easily integrated in existing RL algorithms to learn from active human involvement.

Original: Our method can be generalized across various task settings and human control devices.
Revised: Our method can be generalized across various task settings and human control devices.

Original: The experiments show that the proposed PVP method enables safety guarantees and high training efficiency.
Revised: The experiments show that the proposed PVP method enables safety guarantees and high training efficiency.

Original: Compared to previous HL baselines, PVP achieves better performance with less cost of human involvement.
Revised: Compared to previous HL baselines, PVP achieves better performance with less cost of human involvement.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: [Figure]
Revised: [Figure]

Original: As shown in the left panel of Fig.
Revised: As shown in the left panel of Fig.

Original: 1, reinforcement learning (RL) method learns from the interactions between agent and environment.
Revised: 1, reinforcement learning (RL) method learns from the interactions between agent and environment.

Original: The source of supervision comes from the reward function.
Revised: The source of supervision comes from the reward function.

Original: However, Russell (2019) challenges the existing RL methods that it is hard to comprehensively represent human preferences into a scalar reward function (Dafoe et al., 2020).
Revised: However, Russell (2019) challenges the existing RL methods that it is hard to comprehensively represent human preferences into a scalar reward function (Dafoe et al., 2020).

Original: The manually designed reward function, which might be misaligned with human intent and preference, often leads to undesired agent behaviors (Leike et al., 2018; Krakovna et al., 2020).
Revised: The manually designed reward function, which might be misaligned with human intent and preference, often leads to undesired agent behaviors (Leike et al., 2018; Krakovna et al., 2020).

Original: As a promising complement to RL,
Revised: As a promising complement to RL,

Original: Human-in-the-loop Learning (HL) can overcome costly reward engineering and convey human intent and preference to the learning process directly through human involvement.
Revised: Human-in-the-loop Learning (HL) can overcome costly reward engineering and convey human intent and preference to the learning process directly through human involvement.

Original: HL with Passive Human Involvement.
Revised: HL with Passive Human Involvement.

Original: As shown in the middle panel of Fig.
Revised: As shown in the middle panel of Fig.

Original: 1, many works learn from passive human involvement.
Revised: 1, many works learn from passive human involvement.

Original: The neural policy operates the robot and the human subjects can provide demonstration directly upon request of the learning agents (Mandel et al., 2017; Menda et al., 2019; Jonnavittula & Losey, 2021).
Revised: The neural policy operates the robot and the human subjects can provide demonstration directly upon request of the learning agents (Mandel et al., 2017; Menda et al., 2019; Jonnavittula & Losey, 2021).

Original: This exposes human subjects to potential risks since they do not fully control the system.
Revised: This exposes human subjects to potential risks since they do not fully control the system.

Original: Alternatively, human subjects can evaluate the collected trajectories after the agent-environment interaction (Christiano et al., 2017; Guan et al., 2021;
Revised: Alternatively, human subjects can evaluate the collected trajectories after the agent-environment interaction (Christiano et al., 2017; Guan et al., 2021;

Original: Reddy et al., 2018; Warnell et al., 2018; Sadigh et al., 2017; Palan et al., 2019).
Revised: Reddy et al., 2018; Warnell et al., 2018; Sadigh et al., 2017; Palan et al., 2019).

Original: These methods can be applied to the tasks that human can not conduct, such as moving a six-legged Ant robot via providing exact torque at each joint (Christiano et al., 2017).
Revised: These methods can be applied to the tasks that human can not conduct, such as moving a six-legged Ant robot via providing exact torque at each joint (Christiano et al., 2017).

Original: However, for those tasks that human can demonstrate, these methods are unable to have real-time feedback from human subject during agent-environment interaction.
Revised: However, for those tasks that human can demonstrate, these methods are unable to have real-time feedback from human subject during agent-environment interaction.

Original: HL with Active Human Involvement.
Revised: HL with Active Human Involvement.

Original: As shown in the right panel of Fig.
Revised: As shown in the right panel of Fig.

Original: 1, human subjects can proactively participate based on their own judgment.
Revised: 1, human subjects can proactively participate based on their own judgment.

Revised: There are many works that utilize the idea of learning from advice.

Original: They can decide to terminate the episode if a near-accidental situation happens (Zhang & Cho, 2017; Abel et al., 2017; Saunders et al., 2018;
Revised: TAMER framework (Knox & Stone, 2012) learns policy from human-provided evaluative feedback (human-generated reward). The Task-Instruction-Contingency-Shaping (TISC) method (Najar et al., 2020) also uses evaluative feedback for accelerating the learning. Evaluative feedback is a boolean criticizing correct/wrong. This is similar to the intervention in our framework.

Revised: However, in TICS, humans provide high-level instructions, e.g. pointing to the left/right, while in

Original:  Xu et al., 2022).
Revised: PVP humans provide intervention and low-level demonstrations. TICS directly modifies the policy action distribution based on human feedback, while PVP uses a proxy value to indirectly encourage agents to follow human preference. Other works allow human subjects to decide to terminate the episode if a near-accidental situation happens (Zhang & Cho, 2017; Abel et al., 2017; Saunders et al., 2018; Xu et al., 2022).

Original: Recent studies explore active human involvement methods through intervention and demonstration in the human-robot shared autonomy (Menda et al., 2019;Kelly et al., 2019;
Revised: Recent studies explore active human involvement methods through intervention and demonstration in the human-robot shared autonomy (Menda et al., 2019;

Original:  Spencer et al., 2020; Li et al., 2022b; Jonnavittula & Losey, 2021;
Revised: Kelly et al., 2019; Spencer et al., 2020; Li et al., 2022b; Jonnavittula & Losey, 2021;

Original: Xu et al., 2022).
Revised: Xu et al., 2022).

Original: However, previous methods do not fully utilize the power of human involvement.
Revised: However, previous methods do not fully utilize the power of human involvement.

Original: Kelly et al.
Revised: Kelly et al.

Original: (2019) do not leverage data collected by agents, while Mandlekar et al.
Revised: (2019) do not leverage data collected by agents, while Mandlekar et al.

Original: (2020) do not suppress undesired actions likely intervened by human.
Revised: (2020) do not suppress undesired actions likely intervened by human.

Original: Meanwhile, Spencer et al.
Revised: Meanwhile, Spencer et al.

Original: (2020) and Mandlekar et al.
Revised: (2020) and Mandlekar et al.

Original: (2020) focus on optimizing actions step-wise without considering the temporal consistency between steps.
Revised: (2020) focus on optimizing actions step-wise without considering the temporal consistency between steps.

Original: These drawbacks harm the learning efficiency and thus incur more human involvement.
Revised: These drawbacks harm the learning efficiency and thus incur more human involvement.

Original: Moreover, previous methods lack experiments to evaluate the generalizability to different task settings and human control devices.
Revised: Moreover, previous methods lack experiments to evaluate the generalizability to different task settings and human control devices.

Original: Another challenge for RL methods is to learn from the data collected from human-robot shared control since the data is drastically off-policy with significant distribution shift.
Revised: Another challenge for RL methods is to learn from the data collected from human-robot shared control since the data is drastically off-policy with significant distribution shift.

Original: One solution is to frame active human involvement as a form of interactive imitation learning, where the human demonstration is used to train the agent and the data generated by the learning agent is discarded (Mandlekar et al., 2020; Kelly et al., 2019).
Revised: One solution is to frame active human involvement as a form of interactive imitation learning, where the human demonstration is used to train the agent and the data generated by the learning agent is discarded (Mandlekar et al., 2020; Kelly et al., 2019).

Original: The information in the agent exploration, e.g. the latent model of state transitions (Levine et al., 2020), is lost.
Revised: The information in the agent exploration, e.g. the latent model of state transitions (Levine et al., 2020), is lost.

Original: Alternative is to use the intervention signals to train an intervention predictor to block dangerous actions and moderate the stress of human subjects (Kelly et al., 2019; Abel et al., 2017; Saunders et al., 2018).
Revised: Alternative is to use the intervention signals to train an intervention predictor to block dangerous actions and moderate the stress of human subjects (Kelly et al., 2019; Abel et al., 2017; Saunders et al., 2018).

Original: Reward learning can be used to extract knowledge from human demonstration and intervention (Ibarz et al., 2018;
Revised: Reward learning can be used to extract knowledge from human demonstration and intervention (Ibarz et al., 2018;

Original: Wang et al., 2018).
Revised: Wang et al., 2018).

Original: Offline
Revised: Offline

Original: RL method is used to learn from human-robot mixed data (Peng et al., 2021;
Revised: RL method is used to learn from human-robot mixed data (Peng et al., 2021;

Original: Li et al., 2022b).
Revised: Li et al., 2022b).

Original: In this work, we build upon the existing RL framework and relabel the data collected from shared control with proxy values, connecting RL with active human involvement.
Revised: In this work, we build upon the existing RL framework and relabel the data collected from shared control with proxy values, connecting RL with active human involvement.

Original:  P ROBLEM F ORMULATION
Revised:  P ROBLEM F ORMULATION

Original: Policy learning aims at finding a policy to solve the sequential decision-making task, which is usually modeled by a Markov decision process (MDP).
Revised: Policy learning aims at finding a policy to solve the sequential decision-making task, which is usually modeled by a Markov decision process (MDP).

Original: MDP is defined by the tuple M = ⟨S , A , P ,R,γ,d 0 ⟩ consisting of a state space S , an action space A , a state transition function P ∶ S
Revised: MDP is defined by the tuple M = ⟨S , A , P ,R,γ,d 0 ⟩ consisting of a state space S , an action space A , a state transition function P ∶ S

Original: ×
Revised: ×

Original: A → S , a reward function R ∶ S × A →
Revised: A → S , a reward function R ∶ S × A →

Original: [ R min ,R max ] , a discount factor γ ∈ ( 0 , 1 ) , and an initial state distribution d 0 ∶ S → [ 0 , 1 ] .
Revised: [ R min ,R max ] , a discount factor γ ∈ ( 0 , 1 ) , and an initial state distribution d 0 ∶ S → [ 0 , 1 ] .

Original: The goal of conventional reinforcement learning is to learn a novice policy π n ( a ∣ s )
Revised: The goal of conventional reinforcement learning is to learn a novice policy π n ( a ∣ s )

Original: ∶ S × A → [ 0 , 1 ] that can maximize the expected cumulative return: 
Revised: ∶ S × A → [ 0 , 1 ] that can maximize the expected cumulative return: π n = arg max π n E τ ∼ P

Original: [Equation]
Revised: πn

Original: [∑ Tt = 0 γ t r ( s t ,a t )] , wherein τ = ( s 0 ,a 0 ,...,s T ,a T ) is the trajectory sam- pled from trajectory distribution P π n induced by π n , d 0 and P .
Revised: [∑ Tt = 0 γ t r ( s t ,a t )] , wherein τ = ( s 0 ,a 0 ,...,s T ,a T ) is the trajectory sam- pled from trajectory distribution P π n induced by π n , d 0 and P .

Original: Here π n defines a stochastic policy, while deterministic policy can be denoted as µ n ( s ) ∶ S
Revised: Here π n defines a stochastic policy, while deterministic policy can be denoted as µ n ( s ) ∶ S

Original: → A and its action distribution can be described by a Dirac delta distribution π n ( a ∣ s ) =
Revised: → A and its action distribution can be described by a Dirac delta distribution π n ( a ∣ s ) =

Original: δ ( a − µ n ( s )) .
Revised: δ ( a − µ n ( s )) .

Original: The reward function imposes an assumption that the reward can fully reflect the intentions of the users and incentivize the desired behaviors of the agents.
Revised: The reward function imposes an assumption that the reward can fully reflect the intentions of the users and incentivize the desired behaviors of the agents.

Original: However, the assumption may not always hold and the learned agent may obtain biased behaviors or figure out the loophole to finish the task (Russell, 2019; Leike et al., 2018).
Revised: However, the assumption may not always hold and the learned agent may obtain biased behaviors or figure out the loophole to finish the task (Russell, 2019; Leike et al., 2018).

Original: Revisiting the primal goal when developing learning systems, we find the reward is not a necessity since what we really want to achieve is the realization of human preference in the learned behaviors and, as suggested by Russell (2019), the ultimate source of information about human preferences is human behavior .
Revised: Revisiting the primal goal when developing learning systems, we find the reward is not a necessity since what we really want to achieve is the realization of human preference in the learned behaviors and, as suggested by Russell (2019), the ultimate source of information about human preferences is human behavior .

Original: We thus study the reward-free setting and incorporate real human subjects into the training loop.
Revised: We thus study the reward-free setting and incorporate real human subjects into the training loop.

Original: During training, a human subject accompanies the novice policy and can intervene the agent by taking over the control to demonstrate correct behaviors.
Revised: During training, a human subject accompanies the novice policy and can intervene the agent by taking over the control to demonstrate correct behaviors.

Original: As discussed in Sec. 4.3, we show that such active human involvement can ensure the safety of human-robot system.
Revised: As discussed in Sec. 4.3, we show that such active human involvement can ensure the safety of human-robot system.

Original: We formulate this setting by assuming human subject has a human policy π h ( a h ∣ s )
Revised: We formulate this setting by assuming human subject has a human policy π h ( a h ∣ s )

Original: ∶ S × A → [ 0 , 1 ] , which outputs human action a h ∈ A at each time step.
Revised: ∶ S × A → [ 0 , 1 ] , which outputs human action a h ∈ A at each time step.

Original: We model the human subjects’ intervention behaviors by intervention policy I (⋅∣ s,a n ) .
Revised: We model the human subjects’ intervention behaviors by intervention policy I (⋅∣ s,a n ) .

Original: In earlier methods such as DAgger (Ross & Bagnell, 2010), the intervention policy is a Bernoulli distribution and the control authority switches back and forth between the novice and the expert.
Revised: In earlier methods such as DAgger (Ross & Bagnell, 2010), the intervention policy is a Bernoulli distribution and the control authority switches back and forth between the novice and the expert.

Original: Later studies allow the human subjects to intervene and take full control (Wang et al., 2018; Xu et al., 2022;
Revised: Later studies allow the human subjects to intervene and take full control (Wang et al., 2018; Xu et al., 2022;

Original: Saunders et al., 2018; Li et al., 2022b).
Revised: Saunders et al., 2018; Li et al., 2022b).

Original: In this case the intervention policy can be considered as a deterministic policy denoted by I ( s,a n ) ∶ S
Revised: In this case the intervention policy can be considered as a deterministic policy denoted by I ( s,a n ) ∶ S

Original: × A → { 0 , 1 } where a n ∼ π n (⋅∣ s ) is agent’s action.
Revised: × A → { 0 , 1 } where a n ∼ π n (⋅∣ s ) is agent’s action.

Original: With the intervention policy, we can summarize the behavior policy π b that generates actions applied to the environment as:
Revised: With the intervention policy, we can summarize the behavior policy π b that generates actions applied to the environment as:

Original: [Equation]
Revised: [Equation]

Original: With such a model of active human involvement, we can now formulate our goal.
Revised: With such a model of active human involvement, we can now formulate our goal.

Original: Recall that our primal goal is to find the novice agents whose behaviors are compliant to human preferences.
Revised: Recall that our primal goal is to find the novice agents whose behaviors are compliant to human preferences.

Original: To quantitatively measure the extent where agents accomplish this, we consider two metrics and thus form a twofold problem.
Revised: To quantitatively measure the extent where agents accomplish this, we consider two metrics and thus form a twofold problem.

Original: On one hand, task-specified metrics are available in our experiments since we need to evaluate the performance of the learned agent when they are running independently without human involvement in testing time.
Revised: On one hand, task-specified metrics are available in our experiments since we need to evaluate the performance of the learned agent when they are running independently without human involvement in testing time.

Original: These metrics can be the success rate in navigation or the scores in games.
Revised: These metrics can be the success rate in navigation or the scores in games.

Original: Though agents do not have access to the metrics, human subjects are aware of how these metrics are computed and the correspondence between the metrics and the success of the tasks.
Revised: Though agents do not have access to the metrics, human subjects are aware of how these metrics are computed and the correspondence between the metrics and the success of the tasks.

Original: Therefore, we can use these metrics as the measurements to evaluate the realization of human preference:
Revised: Therefore, we can use these metrics as the measurements to evaluate the realization of human preference:

Original: Problem 1 (Preference Compliance) .
Revised: Problem 1 (Preference Compliance) .

Original: Find a novice policy that maximizes the underlying taskspecified metric U :π ∗ n = arg max π n E τ ∼ P
Revised: Find a novice policy that maximizes the underlying taskspecified metric U :

Original: πn [ U ( τ )] .
Revised: [Equation]

Original: Note that the task-specified metric is not accessible to the learning agent and thus can not be optimized directly.
Revised: Note that the task-specified metric is not accessible to the learning agent and thus can not be optimized directly.

Original: Instead, human intervention I ( s,a ) and demonstration a h ∼ π h (⋅∣ s ) are the only sources of supervision in our current reward-free setting.
Revised: Instead, human intervention I ( s,a ) and demonstration a h ∼ π h (⋅∣ s ) are the only sources of supervision in our current reward-free setting.

Original: In active human involvement, human subjects can intervene at any time at their discretion.
Revised: In active human involvement, human subjects can intervene at any time at their discretion.

Original: The most common cases of human involvement are the near-accidental situations.
Revised: The most common cases of human involvement are the near-accidental situations.

Original: It is also possible for the human subjects to intervene if the agents perform poorly.
Revised: It is also possible for the human subjects to intervene if the agents perform poorly.

Original: Inversely, if the human subjects do not involve, then current states and agent’s actions are deemed to be human-compatible.
Revised: Inversely, if the human subjects do not involve, then current states and agent’s actions are deemed to be human-compatible.

Original: Therefore, we summarize another problem that need to be resolved by our method:
Revised: Therefore, we summarize another problem that need to be resolved by our method:

Original: Problem 2 (Involvement Minimization) .
Revised: Problem 2 (Involvement Minimization) .

Original: Find a novice policy that minimizes the human involvement: π n = arg min π n E τ ∼ P πb
Revised: Find a novice policy that minimizes the human involvement: π n = arg min π n E τ ∼ P πb

Original: [ I ( s,a )] .
Revised: [ I ( s,a )] .

Original: In next section, we will discuss our insights and how we build a concise, general, and efficient learning method to address the aforementioned problems.
Revised: In next section, we will discuss our insights and how we build a concise, general, and efficient learning method to address the aforementioned problems.

Original:  M ETHOD
Revised:  M ETHOD

Original: We propose the Proxy Value Propagation (PVP) method to tackle the problems discussed in the previous section.
Revised: We propose the Proxy Value Propagation (PVP) method to tackle the problems discussed in the previous section.

Original: PVP turns a value-based RL method into an efficient human-in-the-loop policy learning method that learns from active human involvement.
Revised: PVP turns a value-based RL method into an efficient human-in-the-loop policy learning method that learns from active human involvement.

Original: PVP is compatible with various task settings such as continuous and discrete action spaces and various human control devices.
Revised: PVP is compatible with various task settings such as continuous and discrete action spaces and various human control devices.

Original: In this section, we first recap the basic workflow of value-based RL method.
Revised: In this section, we first recap the basic workflow of value-based RL method.

Original: Based on that, we introduce the motivation and the design of PVP.
Revised: Based on that, we introduce the motivation and the design of PVP.

Original: Finally, we describe the implementation details.
Revised: Finally, we describe the implementation details.

Original: Value-based Reinforcement Learning.
Revised: Value-based Reinforcement Learning.

Original: In the conventional RL method, the state-action value and state value of policy π are defined as Q ( s,a ) =
Revised: In the conventional RL method, the state-action value and state value of policy π are defined as Q ( s,a ) =

Original: E [∑ ∞ t = 

Revised: E [∑ ∞ t =

Original: γ t r ( s t ,a t )] and V ( s ) =
Revised: γ t r ( s t ,a t )] and V ( s ) =

Original: E a ∼ π (⋅∣ s ) Q ( s,a ) , respectively.
Revised: E a ∼ π (⋅∣ s ) Q ( s,a ) , respectively.

Original: The optimal policy is expected to maximize the cumulative return J ( π ) =
Revised: The optimal policy is expected to maximize the cumulative return J ( π ) =

Original: E s ∼ d 0 V ( s ) .
Revised: E s ∼ d 0 V ( s ) .

Original: A neural network is commonly used to estimate the value function with Bellman backup: Q ( s,a ) ←
Revised: A neural network is commonly used to estimate the value function with Bellman backup: Q ( s,a ) ←

Original: R ( s,a )+ γ max a ′ Q ( s ′ ,a ′ ) , where s ′ is the next state.
Revised: R ( s,a )+ γ max a ′ Q ( s ′ ,a ′ ) , where s ′ is the next state.

Original: To learn the value network, stochastic gradient descent on the temporal difference (TD) loss is conducted:
Revised: To learn the value network, stochastic gradient descent on the temporal difference (TD) loss is conducted:

Original: [Equation]
Revised: [Equation]

Original: If we discard the instant reward, the Q value is optimized to fit Q ( s,a ) ← γ max a ′ Q ( s ′ ,a ′ ) .
Revised: If we discard the instant reward, the Q value is optimized to fit Q ( s,a ) ← γ max a ′ Q ( s ′ ,a ′ ) .

Original: Based on the learned value function, the policy π θ parameterized by θ can be learned by optimizing
Revised: Based on the learned value function, the policy π θ parameterized by θ can be learned by optimizing

Original: [Equation]
Revised: [Equation]

Original: 4. P ROXY V ALUE P ROPAGATION
Revised: 4. P ROXY V ALUE P ROPAGATION

Original: The major challenge to efficiently learn from active human involvement is the scarcity of information.
Revised: The major challenge to efficiently learn from active human involvement is the scarcity of information.

Original: To address the issue, we introduce a key observation and the goals behind the design of our method.
Revised: To address the issue, we introduce a key observation and the goals behind the design of our method.

Original: The observation is that when a human subject intervenes in the human-robot shared control process, it leads to a clear signal that human subject is not satisfied with the current performance of the agent, either due to its unsafe actions or poorly performing behaviors.
Revised: The observation is that when a human subject intervenes in the human-robot shared control process, it leads to a clear signal that human subject is not satisfied with the current performance of the agent, either due to its unsafe actions or poorly performing behaviors.

Original: Based on this, we can summarize that the learned policy should (1) replicate the behaviors demonstrated by the human subject and (2) avoid performing actions that are once intervened by the human subject .
Revised: Based on this, we can summarize that the learned policy should (1) replicate the behaviors demonstrated by the human subject and (2) avoid performing actions that are once intervened by the human subject .

Original: The requirement can be fulfilled by the value-based RL framework.
Revised: The requirement can be fulfilled by the value-based RL framework.

Original: A deeper look at Eq.
Revised: A deeper look at Eq.

Original: 4 suggests an optimal deterministic policy will always choose actions with the largest Q values.
Revised: 4 suggests an optimal deterministic policy will always choose actions with the largest Q values.

Original: Therefore we can manipulate the Q values to induce desired behaviors:Goal 1 (Replicate Human Actions and Suppress Undesired Actions) . If human subject intervenes at some states, the human actions a h ∼ π h should always have higher values than other actions in these states.
Revised: Therefore we can manipulate the Q values to induce desired behaviors: If human subject intervenes at some states, the human actions a h ∼ π h should always have higher values than other actions in these states.

Original: At the same time, the agent actions a n ∼ π n should always have lower values than others since they are rejected by human subject.
Revised: At the same time, the agent actions a n ∼ π n should always have lower values than others since they are rejected by human subject.

Original: To implement Goal 1, we overwrite the Q value of the human action a h during human intervention to be + 1 and the novice action a n to be − 1 .Such idea can be implemented via the Proxy Value loss:
Revised: To implement this idea, we overwrite the Q value of the human action a h during human intervention to be + 1 and the novice action a n to be − 1 .

Original: [Equation][Figure]
Revised: [Equation]

Original: Now turn to the states where human subjects have not intervened before.

Original: An intuition is that we should select actions at these states that lead to the transitions to those states where human have previously intervened (human-involved states).

Original: By doing so we reduce the problem to a previously solved one and then can replicate the actions previously taken by human, which are assumed to be optimal if Goal 1 stands.

Original: How to drive agents to human-involved states?

Original: We have discussed in Goal 1 that high values action will be chosen by the policy.

Original: As shown in Fig.

Original: 2, we find that if we let the action athat leads to human-involved state s h to have higher value than other actions e.g. a 2 , then the policy will automatically choose a 1 and transit to s h .

Original: A natural idea to automatically label action a 1 with high value is to reuse the TD learning.

Original: When minimizing TD error following Eq. 3, the maximal Q values in next state will be backed up to prior states.

Original: Illustrated in Fig.

Original: 2, let’s consider in prior state

Original: s we have two actions a 1 ∶ P( s,a 1 ) = s h and a 2 ∶ P( s,a 2 ) =
Revised: [Equation]

Original: s n , leading to human-involved and non-human-involved states respectively.

Original: We can find that max a Q ( s h ,a ) > max a Q ( s n ,a ) if Goal 1 stands since Q ( s h ,a h ) will have high value.

Original: Then an optimal policy after TD learning will choose µ ∗ ( s )

Original: = a 1 since the high value from s h is propagated to state s and makes Q ( s,a 1 ) have high value too.

Original: This leads to our second design goal: Goal 2 (Reach Human-involved States) . We should propagate the values of human-involved states to prior states, so that the optimal policy will learn actions that reach human-involved states. Considering Goal 1 and Goal 2 together, the loss to update Q network thus becomes:
Revised: To propagate the information of those desired actions provided by human subjects, we combine TD loss to update Q network:

Original: [Equation]
Revised: [Equation]

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: Base RL Methods.
Revised: Base RL Methods.

Original: Our method can be implemented for both continuous and discrete action spaces.
Revised: Our method can be implemented for both continuous and discrete action spaces.

Original: In continuous action space, we extend TD3 (Fujimoto et al., 2018) with the balanced buffer and
Revised: In continuous action space, we extend TD3 (Fujimoto et al., 2018) with the balanced buffer and

Original: PV loss.
Revised: PV loss.

Original: In discrete action space, we use DQN (Mnih et al., 2015).
Revised: In discrete action space, we use DQN (Mnih et al., 2015).

Original: In order to verify our idea in a minimalist and concise framework, though compatible, we do not apply advanced techniques to
Revised: In order to verify our idea in a minimalist and concise framework, though compatible, we do not apply advanced techniques to

Original: DQN such as prioritized replay buffer (Schaul et al., 2015), double Q learning (Van Hasselt et al., 2016) and ensemble method (Osband et al., 2016).
Revised: DQN such as prioritized replay buffer (Schaul et al., 2015), double Q learning (Van Hasselt et al., 2016) and ensemble method (Osband et al., 2016).

Original: While TD3 uses deterministic policy, DQN adopts epsilon-greedy exploration that makes the policy stochastic.
Revised: While TD3 uses deterministic policy, DQN adopts epsilon-greedy exploration that makes the policy stochastic.

Original: We remove the action noise in DQN and simply follow the argmax rule to select actions: µ ∗ ( s ) =
Revised: We remove the action noise in DQN and simply follow the argmax rule to select actions: µ ∗ ( s ) =

Original: arg max a Q ( s,a ) .
Revised: arg max a Q ( s,a ) .

Original: The primary reason is that according to the feedback of human subjects, stochastic agents make human subjects experience excessive fatigue since it is hard to monitor and correct agents’ noisy actions.Removing action noise greatly improves the well-being of human subjects.
Revised: The primary reason is that according to the feedback of human subjects, stochastic agents make human subjects experience excessive fatigue since it is hard to monitor and correct agents’ noisy actions.

Original: Balanced Buffers.
Revised: Balanced Buffers.

Original: The intervention gradually becomes sparse as the agent learns to reduce human intervention.
Revised: The intervention gradually becomes sparse as the agent learns to reduce human intervention.

Original: However, those sparse intervention signals contain even more important information on how to behave under critical situations.
Revised: However, those sparse intervention signals contain even more important information on how to behave under critical situations.

Original: Previous method (Li et al., 2022b) stores agent data and human data into one buffer and samples them uniformly.
Revised: Previous method (Li et al., 2022b) stores agent data and human data into one buffer and samples them uniformly.

Original: The human demonstrations are overwhelmed by the amount of agent-generated trajectories, leading to inefficient learning of critical human behaviors.
Revised: The human demonstrations are overwhelmed by the amount of agent-generated trajectories, leading to inefficient learning of critical human behaviors.

Original: For example, the driving policy sometimes fails to master acceleration at the beginning of an episode, even though the human subject has already demonstrated the expected maneuver multiple times.
Revised: For example, the driving policy sometimes fails to master acceleration at the beginning of an episode, even though the human subject has already demonstrated the expected maneuver multiple times.

Original: This is because the demonstration of initial acceleration only lasts a short period of time and thus is scarce in the buffer.
Revised: This is because the demonstration of initial acceleration only lasts a short period of time and thus is scarce in the buffer.

Original: To address this issue, we utilize two replay buffers to store transitions with or without human intervention.
Revised: To address this issue, we utilize two replay buffers to store transitions with or without human intervention.

Original: The human buffer B h = {( s,a n ,a h ,s ′ )} stores data during human involvement and the novice buffer B n = {( s,a n ,s ′ )} stores the transitions during agent’s independent exploration.
Revised: The human buffer B h = {( s,a n ,a h ,s ′ )} stores data during human involvement and the novice buffer B n = {( s,a n ,s ′ )} stores the transitions during agent’s independent exploration.

Original: In each training iteration, we sample two equally-sized batches b n , b h from B n , B h respectively, each has N / 2 samples.
Revised: In each training iteration, we sample two equally-sized batches b n , b h from B n , B h respectively, each has N / 2 samples.

Original: We compute PV loss J PV use b h and concatenate b n and b h to form a batch with N transitions to compute the TD loss J TD .
Revised: We compute PV loss J PV use b h and concatenate b n and b h to form a batch with N transitions to compute the TD loss J TD .

Original: N is the hyper-parameter batch size.
Revised: N is the hyper-parameter batch size.

Original: With such design, our method can balance the data from human and from agent.
Revised: With such design, our method can balance the data from human and from agent.

Original: In the initial acceleration example above, as training goes, the agent gradually masters driving skills, e.g. driving on a straight road or turning.
Revised: In the initial acceleration example above, as training goes, the agent gradually masters driving skills, e.g. driving on a straight road or turning.

Original: After that, human subject will only demonstrate the initial acceleration that the agent does not yet learn.
Revised: After that, human subject will only demonstrate the initial acceleration that the agent does not yet learn.

Original: Those demonstrations will gradually dominate the human buffer until agent masters them.
Revised: Those demonstrations will gradually dominate the human buffer until agent masters them.

Revised: [Figure]

Original: 4. T HEORETIC S AFETY G UARANTEE
Revised: 4. H UMAN I NTENT C OMPLIANCE

Original: In this section, we show that in the setting of active human involvement, the training and test time safety can be bounded by theerror rates of human subjects, namely the probability of making wrong interventions andwrong actions. Improving the state coverage of human-involved states during training can also improve the test time safety. The detailed proof is provided in Appendix D.
Revised: Here we provide a conceptual framework to describe the compliance of human intention. We introduce a simple theorem showing that, in the setting of active human involvement, the compliance of human intention can be grounded by the human policy performance, the accuracy of human intervention and the overall intervention rate.

Original: We define the ground-truth indicator C ∶ S × A → { 0 , 1 } denoting whether the action will lead to unsafe state.

Original: The indicator is determined by the environment and is not revealed to the learning algorithm. We measure safety with the notation of cumulative discounted probability offailure as: [Equation] (7) which indicates the chance of entering dangerous states in current time step as well as in future trajectories deduced by the policy π .
Revised: Theorem 1 (Upper bound of the training risk) .

Original: We use S [Equation] s 0 ) and S π b = E s 0 ∼ d 0 Sπ b ( s 0 ) to denote the safety of human policy and the behavior policy, respectively.
Revised: Let C ∶ S

Original: We introduce two important assumptions on the human subject without imposing any constraint on the human policy. Assumption 1 (Error rate of human policy) . For all states, the step-wise probability of that the human subject produces an unsafe action is bounded by a small value ϵ < 1 : [Equation]
Revised: ×

Original: Assumption 2 (Error rate of intervention policy) . During human-robot shared control, the probability that the human subject does not intervene when the action is unsafe is bounded by a small κ < 1 : [Equation] [Equation]
Revised: A → { 0 , 1 } be the ground-truth indicator of the intention violation i.e. safety violation, saying whether the action is undesired.

Original: Training Safety Guarantee . The main theorem bounds the probability of failure in training time: Theorem 1 (Upper bound oftraining risk) . The expected discounted probability of failure S π b of the behavior policy π b is bounded by the error rate of the human action  ϵ , the error rate ofthe human intervention κ and the intervention rate ψ :
Revised: Denote the upper bound of human action error rate as E a ∼ π h (⋅∣ s ) C ( s,a ) ≤ ϵ , the upper bound of human intervention error rate as E s ∼ P πb ,a ∼ π n (⋅∣ s ) ( 1 − I ( s,a ))

Original: [Equation] 10) wherein ψ = E s ∼ P πb [ 1 − I ( s,µ n ( s ))] denotes the intervention rate. Testing Safety Guarantee .
Revised: C ( s,a ) ≤ κ and the intervention rate as ψ =

Original: We can write an expected novice policy by factorizing the stochasticity of the training process and human subjects into the action distribution:

Original: [Equation]
Revised: E s ∼ P

Original: 11) where M ( s )= 1 denotes human has once intervened in state s . Note that π ∗ ( a ∣ s ) is not an outcome deterministic policy but instead a hypothetical policy for analysis. The test-time safety guarantee is: Theorem 2 (Upper bound of testing risk) .
Revised: πb I ( s,a n ) .

Original: Theexpected cumulative discounted probability of failure
Revised: The discounted intent violation S π b ( s 0 ) =

Original: S π ∗{+E τ ∼ P πb ∑ t = 
 γ t C ( s t ,a t )+} of the expected novice policy π ∗ is bounded bythe error rate of the human policy ϵ , the error rate of the intervention policy κ and human-involved state coverage ξ :
Revised: {+E τ ∼ P πb ∑ t = 
 γ t C ( s t ,a t )+} of the behavior policy π b is bounded by

Original: [Equation]
Revised: [Equation]

Original: wherein ξ = E s ∼ P π ∗ M ( s ) measuring the coverage of human-involved states. [Figure]
Revised: The proof and detailed explanation is provided in Appendix D.

Original:  E XPERIMENTS 5. E XPERIMENTAL S ETTING
Revised:  E XPERIMENTS 5. E XPERIMENTAL S ETTING

Original: Tasks.
Revised: Tasks.

Original: We conduct experiments on various control tasks with different observation and action spaces.
Revised: We conduct experiments on various control tasks with different observation and action spaces.

Original: For continuous action space, we use two driving environments, MetaDrive safety benchmark (Li et al., 2022a) and CARLA Town01 environment (Dosovitskiy et al., 2017).
Revised: For continuous action space, we use two driving environments, MetaDrive safety benchmark (Li et al., 2022a) and CARLA Town01 environment (Dosovitskiy et al., 2017).

Original: In both tasks, the agent needs to steer the target vehicle with low-level acceleration, brake, and steering, to reach the destination.
Revised: In both tasks, the agent needs to steer the target vehicle with low-level acceleration, brake, and steering, to reach the destination.

Original: We use sensory state vector in MetaDrive and the bird-eye view image in CARLA as observation.
Revised: We use sensory state vector in MetaDrive and the bird-eye view image in CARLA as observation.

Original: Note that for MetaDrive task, there exists a split of training and test environments and we present the performance of the learned agent in a held-out test environment.
Revised: Note that for MetaDrive task, there exists a split of training and test environments and we present the performance of the learned agent in a held-out test environment.

Original: For discrete action space, we use
Revised: For discrete action space, we use

Original: MiniGrid Two Room task (Chevalier-Boisvert et al., 2018) and Atari Skiing game (Bellemare et al., 2013).
Revised: MiniGrid Two Room task (Chevalier-Boisvert et al., 2018) and Atari Skiing game (Bellemare et al., 2013).

Original: The Two Room is a task requiring heavy exploration since the agent needs to move toward a door and open the door before reaching the destination.
Revised: The Two Room is a task requiring heavy exploration since the agent needs to move toward a door and open the door before reaching the destination.

Original: The observation of MiniGrid is the semantic map of agent’s local neighborhood.
Revised: The observation of MiniGrid is the semantic map of agent’s local neighborhood.

Original: The Atari game is difficult since the agent needs to learn to output meaningful action based on the RGB observation.
Revised: The Atari game is difficult since the agent needs to learn to output meaningful action based on the RGB observation.

Original: MLP is used for MetaDrive task and CNN is used for others as the feature extractors.
Revised: MLP is used for MetaDrive task and CNN is used for others as the feature extractors.

Original: Please refer to Sec.
Revised: Please refer to Sec.

Original: E in Appendix for more information.
Revised: E in Appendix for more information.

Original: Human Interfaces.
Revised: Human Interfaces.

Original: To examine the generalizability of our method, we leverage multiple control devices: Xbox Wireless Controller (Gamepad), keyboard and Logitech G29 Racing Wheel.
Revised: To examine the generalizability of our method, we leverage multiple control devices: Xbox Wireless Controller (Gamepad), keyboard and Logitech G29 Racing Wheel.

Original: We denote the MetaDrive tasks with three devices as MetaDrive-Gamepad/Keyboard/Wheel.
Revised: We denote the MetaDrive tasks with three devices as MetaDrive-Gamepad/Keyboard/Wheel.

Original: CARLA task uses Wheel only.
Revised: CARLA task uses Wheel only.

Original: In MiniGrid and Atari tasks, we use keyboard to provide discrete control signals.
Revised: In MiniGrid and Atari tasks, we use keyboard to provide discrete control signals.

Original: In all tasks, an key in the device is configured to indicate emergency stop.
Revised: In all tasks, an key in the device is configured to indicate emergency stop.

Original: If any discomfort happens, human subjects can pause or stop the experiment immediately.
Revised: If any discomfort happens, human subjects can pause or stop the experiment immediately.

Original: Ethics statement is provided in Appendix A. As shown in Fig.
Revised: Ethics statement is provided in Appendix A. As shown in Fig.

Original: 3, human subjects can intervene and takeover through control devices and the visualization of environments is shown on the screen for monitoring.
Revised: 2, human subjects can intervene and takeover through control devices and the visualization of environments is shown on the screen for monitoring.

Original: Experimental Details.
Revised: Experimental Details.

Original: We implement most of the code with Stable-Baselines

Revised: We implement most of the code with Stable-Baselines

Original: (Raffin et al., 2021).
Revised: (Raffin et al., 2021).

Original: Training results of various baselines in MetaDrive task are obtained from the open-source code by (Li et al., 2022b).
Revised: Training results of various baselines in MetaDrive task are obtained from the open-source code by (Li et al., 2022b).

Original: The pure RL baselines are repeated 5 times with different random seeds, while other human-in-the-loop methods are repeated fewer times due to limited human resources.
Revised: The pure RL baselines are repeated 5 times with different random seeds, while other human-in-the-loop methods are repeated fewer times due to limited human resources.

Original: In the training  of the human-in-the-loop method, we need real human participation and we do not use any simulated user input.
Revised: In the training [Table] of the human-in-the-loop method, we need real human participation and we do not use any simulated user input.

Original: During testing, there is no any form of human involvement.
Revised: During testing, there is no any form of human involvement.

Original: We run the trained agents in the environment for multiple runs and report the average task-specified metrics in the tables.
Revised: We run the trained agents in the environment for multiple runs and report the average task-specified metrics in the tables.

Original: For each run, since we have many checkpoints during training, we will report the metrics of the checkpoint that achieves the highest E τ ∼ P
Revised: For each run, since we have many checkpoints during training, we will report the metrics of the checkpoint that achieves the highest E τ ∼ P

Original: πn U ( τ ) as the performance of this run in the tables.
Revised: πn U ( τ ) as the performance of this run in the tables.

Original: One human subject participates in each experiment.
Revised: One human subject participates in each experiment.

Original: Experiments with humans are conducted on a local computer with an Nvidia GeForce RTX 3080.
Revised: Experiments with humans are conducted on a local computer with an Nvidia GeForce RTX 3080.

Original: We provide the standard deviation if the experiments are repeated multiple runs in tables and figures.
Revised: We provide the standard deviation if the experiments are repeated multiple runs in tables and figures.

Original: Other hyper-parameters are given in the Appendix G.
Revised: Other hyper-parameters are given in the Appendix G.

Original: Baselines.
Revised: Baselines.

Original: We test four native RL baselines:
Revised: We test four native RL baselines:

Original: PPO (Schulman et al., 2017), SAC (Haarnoja et al., 2018), TD3 (Fujimoto et al., 2018) and DQN (Mnih et al., 2015).
Revised: PPO (Schulman et al., 2017), SAC (Haarnoja et al., 2018), TD3 (Fujimoto et al., 2018) and DQN (Mnih et al., 2015).

Original: RL baselines can access environmental reward.
Revised: RL baselines can access environmental reward.

Original: In MetaDrive Safety Benchmark, -1 penalty will be added to reward when a crash happens.
Revised: In MetaDrive Safety Benchmark, -1 penalty will be added to reward when a crash happens.

Original: Three safe RL baselines Constraint Policy Optimization (CPO) (Achiam et al., 2017), PPO-Lagrangian (Stooke et al., 2020), SAC-Lagrangian (Ha et al., 2020) are evaluated.
Revised: Three safe RL baselines Constraint Policy Optimization (CPO) (Achiam et al., 2017), PPO-Lagrangian (Stooke et al., 2020), SAC-Lagrangian (Ha et al., 2020) are evaluated.

Original: In all baselines above, the reward function and cost function (for Safety Benchmark) are defined by the environment and can be accessed by learning algorithms.
Revised: In all baselines above, the reward function and cost function (for Safety Benchmark) are defined by the environment and can be accessed by learning algorithms.

Original: On the other hand, using human-generated dataset in MetaDrive, provided by (Li et al., 2022b), we evaluate IL methods Behavior Cloning, GAIL (Ho & Ermon, 2016) and [Table] [Table]
Revised: On the other hand, using human-generated dataset in MetaDrive, provided by (Li et al., 2022b), we evaluate IL methods Behavior Cloning, GAIL (Ho & Ermon, 2016) and offline RL method

Original: CQL (Kumar et al., 2020).
Revised: CQL (Kumar et al., 2020).

Original: We also run many human-the-loop methods that learn from active human involvement.
Revised: We also run many human-the-loop methods that learn from active human involvement.

Original: Human-Gated DAgger (HG-DAgger) (Kelly et al., 2019), Intervention Weighted
Revised: Human-Gated DAgger (HG-DAgger) (Kelly et al., 2019), Intervention Weighted

Original: Regression (IWR) (Mandlekar et al., 2020) and Human-AI Copilot Optimization (HACO) (Li et al., 2022b) are tested.
Revised: Regression (IWR) (Mandlekar et al., 2020) and Human-AI Copilot Optimization (HACO) (Li et al., 2022b) are tested.

Revised: Demo Video and Code are included in the supplementary materials.

Revised: [Table]

Original: 5. B ASELINE C OMPARISON
Revised: 5. B ASELINE C OMPARISON

Original: Comparing with RL Counterparts.
Revised: Comparing with RL Counterparts.

Original: Fig.
Revised: Fig.

Original: 4 shows the curves of test-time performance of the agents during training.
Revised: 3 shows the curves of test-time performance of the agents during training.

Original: That is, each point in the curves represents the task-specified measurement of the agent saved at that stage of training, performing actions independently without human involvement.
Revised: That is, each point in the curves represents the task-specified measurement of the agent saved at that stage of training, performing actions independently without human involvement.

Original: In MetaDrive task, our method achieves 350 return in 37K steps.
Revised: In MetaDrive task, our method achieves 350 return in 37K steps.

Original: This takes about one hour in the real-world HL experiment.
Revised: This takes about one hour in the real-world HL experiment.

Original: TD3 baseline fails to achieve comparable results even after 300K steps of training.
Revised: TD3 baseline fails to achieve comparable results even after 300K steps of training.

Original: In CARLA Town01, agents learn to drive within 18K steps (30 minutes) with our method, while TD3 can not solve the task.
Revised: In CARLA Town01, agents learn to drive within 18K steps (30 minutes) with our method, while TD3 can not solve the task.

Original: In MiniGrid tasks, our method successfully solves the tasks while vanilla DQN fails, showing that PVP can learn a solution without complex exploration strategies.
Revised: In MiniGrid tasks, our method successfully solves the tasks while vanilla DQN fails, showing that PVP can learn a solution without complex exploration strategies.

Original: We also show experiments on an easier and a harder MiniGrid environments in Appendix F.5, where PVP greatly improves the learning efficiency.
Revised: We also show experiments on an easier and a harder MiniGrid environments in Appendix F.3, where PVP greatly improves the learning efficiency.

Original: In Atari task, our method does not demonstrate impressive improvement.
Revised: In Atari task, our method does not demonstrate impressive improvement.

Original: We hypothesize this is caused by the difficulty in representation learning.
Revised: We hypothesize this is caused by the difficulty in representation learning.

Original: As a comparison, Ibarz et al.
Revised: As a comparison, Ibarz et al.

Original: (2018) use 6,800 human-labeled episodes, containing 50M environment interactions, to learn a reward function and solve the Atari game.
Revised: (2018) use 6,800 human-labeled episodes, containing 50M environment interactions, to learn a reward function and solve the Atari game.

Original: Active human involvement setting generates insufficient data for learning a good representation.
Revised: Active human involvement setting generates insufficient data for learning a good representation.

Original: Comparing with Human-in-the-loop Baselines.
Revised: Comparing with Human-in-the-loop Baselines.

Original: To ensure a fair comparison, we conduct humanin-the-loop experiments with the same group of human subjects to avoid the performance difference caused by personal difference.
Revised: To ensure a fair comparison, we conduct humanin-the-loop experiments with the same group of human subjects to avoid the performance difference caused by personal difference.

Original: To quantitatively assess the human-compatibility and the well-being of human subjects, we report the safety violation and cognitive cost .
Revised: To quantitatively assess the experience of human subjects, we report the safety violation and cognitive cost .

Original: Safety violation is measured by the number of crashes happening during the training in MetaDrive.
Revised: Safety violation is measured by the number of crashes happening during the training in MetaDrive.

Original: This value reflects how many potential damages the human subject might suffer in training.
Revised: This value reflects how many potential damages the human subject might suffer in training.

Original: The cognitive cost is measured by the total amount of human involvement ( Human Data Usage ) and the ratio of human data over total 
Revised: The cognitive cost is measured by the total amount of human involvement ( Human Data Usage ) and the ratio of human data over total sampled data ( Overall

Original: [Table] sampled data ( Overall Intervention Rate ).
Revised: Intervention Rate ).

Original: It shows how much effort humans need to pay to teach and protect the agents.
Revised: It shows how much effort humans need to pay to teach and protect the agents.

Original: As shown in Table 1, all HL methods we test achieve extremely low safety violations in training compared to RL methods, empirically supporting the safety guarantee of the active human involvement in Sec. 4.3.
Revised: As shown in Table 1, all HL methods we test achieve extremely low safety violations in training compared to RL methods, empirically supporting the safety guarantee of the active human involvement in Sec. 4.3.

Original: Compared to other human-in-the-loop methods, our method costs the lowest human efforts in terms of human data usage and overall intervention rate, while greatly outperforms baselines in testing performance.
Revised: Compared to other human-in-the-loop methods, our method costs the lowest human efforts in terms of human data usage and overall intervention rate, while greatly outperforms baselines in testing performance.

Original: Since we are in MetaDrive task with training and test set split, the results suggest PVP can learn high quality agents with generalizability.
Revised: Since we are in MetaDrive task with training and test set split, the results suggest PVP can learn high quality agents with generalizability.

Original: Table 2 presents the final performance in CARLA Town01.
Revised: Table 2 presents the final performance in CARLA Town01.

Original: Compared to RL baselines, our method achieves a decent success rate and route completion rate even though we only utilize 18K environmental interactions.
Revised: Compared to RL baselines, our method achieves a decent success rate and route completion rate even though we only utilize 18K environmental interactions.

Original: In Fig.
Revised: In Fig.

Original: 5, we compare the intervention frequency of PVP and HACO.
Revised: 5, we compare the intervention frequency of PVP and HACO.

Original: It is the number of the human involvement segments in each episode divided by the episode length.
Revised: It is the number of the human involvement segments in each episode divided by the episode length.

Original: Intervention frequency is not equivalent to intervention rate in that it measures how frequently the human subjects involve, reflecting the mental stress human subjects bear during shared control.
Revised: Intervention frequency is not equivalent to intervention rate in that it measures how frequently the human subjects involve, reflecting the mental stress human subjects bear during shared control.

Original: Fig.suggests HACO requires more human demonstration fragments than PVP.
Revised: Fig.suggests HACO requires more human demonstration fragments than PVP.

Original: The drawback stems from that HACO utilizes SAC method with stochastic policy for better exploration.
Revised: The drawback stems from that HACO utilizes SAC method with stochastic policy for better exploration.

Original: The actions produced by the novice policy varies drastically due to randomness, making human subjects stressful to actively intervene.
Revised: The actions produced by the novice policy varies drastically due to randomness, making human subjects stressful to actively intervene.

Original: A detailed visualization of the trajectories generated in human-robot shared control is available in Appendix F.3, clearly illustrating PVP method can generate smoother trajectories. More discussion on the difference between PVP and HACOis presented in Appendix F.2.
Revised: A detailed visualization of the trajectories generated in human-robot shared control is available in Appendix F.2, clearly illustrating PVP method can generate smoother trajectories. 5. C OMPARICNG PVP WITH HACO

Original: The Impact of Input Devices .
Revised: Different Input Devices.

Original: In Appendix F.1, we present the  results with different input devices in MetaDrive benchmark.
Revised: Table 3 presents the experiment results with different input devices in MetaDrive benchmark.

Original: In both settings, the agents trained by PVP outperform those by HACO, showing the generalizability of PVP on different input devices. 
Revised: In both settings, the agents trained by PVP outperform those by HACO, showing the generalizability of PVP on different input devices. [Table] [Figure]

Revised: [Figure]

Revised: Comparing PVP with HACO.

Revised: In Table

Original: Ablation Study.
Revised: [Figure] we observe that HACO (Li et al., 2022b) has performance discrepancy with different input devices. When using Gamepad, human subjects tend to push and pull the stick to the limits, producing extreme values. As we will discuss later, extreme actions are particularly harmful to HACO.

Original: In Appendix F.4, we present the ablation studies to show the importance of TD learning and balanced buffers.
Revised: When using keyboard, the human subjects press arrow keys to indicate increasing/decreasing current steering/acceleration values for an increment. Therefore there will be fewer extreme values happening than using a Gamepad, which explains why the baseline HACO performs better with the keyboard compared to Gamepad. To explain why HACO is vulnerable to extreme action values, we compare the proxy Q values in our method and in HACO baseline in Fig. 4 and find that HACO has much larger magnitude in its proxy values compared to PVP.

Original: Disabling TD learning via setting J TD (  Q  ) = 0 significantly damages the performance of PVP, suggesting that the Goal 2 is critical to the success of PVP.
Revised: This is because HACO updates the state-action values based on CQL objective (Kumar et al., 2020): min Q ( s,a n ) − Q ( s,a h ) . Since extreme values happen frequently when human intervenes through Gamepad, the proxy values in those human actions are reinforced without bound, making the novice policy rapidly learn those actions. In contrast, the proxy values learned via PVP are much more moderate. The novice Q has distinct negative value.

Original: We also find that disabling balanced buffers makes the training unstable and leads to poor performance.
Revised: In contrast, the values of behavior actions, the actions that satisfy human, have positive values. This result reveals the problem of unbounded values in the previous method. Our method instead resolves this issue and has bounded proxy values, leading to stable training. 5. A BLATION S TUDY

Original: Demo Video and Code are included in  the supplementary materials.
Revised: We conduct ablation studies to show the importance of TD learning and balanced buffers. As shown in Table 1, disabling TD learning via setting J TD ( Q ) = 0 significantly damages the performance of PVP, suggesting that propagating information from human-involved states to other states is critical to the success of PVP. We find that reward has no significant impact on the learning performance.

Original: Appendix B describes the structure of the demo video.
Revised: In Table 2, we find that disabling balanced buffers makes the training unstable and leads to poor performance. We also implement PVP based on Soft Actor-critic (Haarnoja et al., 2018) so that the novice policy is now a stochastic policy. The human subjects report that the novice agents in PVP w/ Stochastic Policy oscillate frequently, making them hard to respond when the agents suddenly drive toward the sideroad. We can find that PVP w/ Stochastic Policy performs worse compared to the deterministic version of PVP.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Learning through active human involvement is a promising approach to provide safe and efficient policy learning.
Revised: Learning through active human involvement is a promising approach to provide safe and efficient policy learning.

Original: In this work, we propose Proxy Value Propagation (PVP) that can effectively learn from active human involvement.
Revised: In this work, we propose Proxy Value Propagation (PVP) that can effectively learn from active human involvement.

Original: PVP is applicable to existing value-based RL methods and induces reward-free policy learning.
Revised: PVP is applicable to existing value-based RL methods and induces reward-free policy learning.

Original: Experiments show the superior performance and low human cost of the proposed method in a wide range of tasks, action spaces, and human control devices.
Revised: Experiments show the superior performance and low human cost of the proposed method in a wide range of tasks, action spaces, and human control devices.

Original: Limitations.
Revised: Limitations.

Original: First, the Atari results show that our method has a defect in learning representations of RGB frames.
Revised: First, the Atari results show that our method has a defect in learning representations of RGB frames.

Original: Introducing self-supervised objective is a way to improve representation learning (Srinivas et al., 2020) especially when the data is insufficient as in our method.
Revised: Introducing self-supervised objective is a way to improve representation learning (Srinivas et al., 2020) especially when the data is insufficient as in our method.

Original: Second, we only apply our method to two value-based baselines.
Revised: Second, we only apply our method to two value-based baselines.

Original: Many advanced techniques including exploration encouraging (Osband et al., 2016), prioritized replay buffer (Schaul et al., 2015) and double Q learning (Van Hasselt et al., 2016) can be added upon our method.
Revised: Many advanced techniques including exploration encouraging (Osband et al., 2016), prioritized replay buffer (Schaul et al., 2015) and double Q learning (Van Hasselt et al., 2016) can be added upon our method.

Original: Third, our method is not applicable to the tasks where humans can not provide demonstration, e.g. controlling a six-legged robot via torque at each joint.
Revised: Third, our method is not applicable to the tasks where humans can not provide demonstration, e.g. controlling a six-legged robot via torque at each joint.

Original: A E THICS S TATEMENT
Revised: A E THICS S TATEMENT

Original: Human subjects get paid to participate in the experiments.
Revised: Human subjects get paid to participate in the experiments.

Original: They can pause or stop the experiment if any discomfort happens.
Revised: They can pause or stop the experiment if any discomfort happens.

Original: No human subjects are injured because all tasks we test are in virtual simulation.
Revised: No human subjects are injured because all tasks we test are in virtual simulation.

Original: Each experiment will not last longer than one hour and subjects will rest at least three hours after one experiment.
Revised: Each experiment will not last longer than one hour and subjects will rest at least three hours after one experiment.

Original: During training and data processing, no personal information is revealed in the collected dataset or the trained agents.
Revised: During training and data processing, no personal information is revealed in the collected dataset or the trained agents.

Original: B D EMO V IDEO
Revised: B D EMO V IDEO

Original: Please find our demo video in the supplementary material.
Revised: Please find our demo video in the supplementary material.

Original: This video shows the footage of human experiments and the comparisons between agents learned by the baselines and the proposed method.
Revised: This video shows the footage of human experiments and the comparisons between agents learned by the baselines and the proposed method.

Original: The video contains three sections:
Revised: The video contains three sections:

Original: The first section shows how we learn driving policy in CARLA task within 20 minutes.
Revised: The first section shows how we learn driving policy in CARLA task within 20 minutes.

Original: We also compare the behavior of agents learned from PVP and TD3 baseline.
Revised: We also compare the behavior of agents learned from PVP and TD3 baseline.

Original: In the second section, we show the footage of MetaDrive human experiment where the human subject uses a gamepad as input device and demonstrate the behavior comparison between PVP and TD3 baseline.
Revised: In the second section, we show the footage of MetaDrive human experiment where the human subject uses a gamepad as input device and demonstrate the behavior comparison between PVP and TD3 baseline.

Original: In the third section, we show the applicability of our method in discrete control tasks.
Revised: In the third section, we show the applicability of our method in discrete control tasks.

Original: The behavior comparison between PVP and DQN baseline in MiniGrid Empty Room and Four Room and Atari Skiing tasks are provided.
Revised: The behavior comparison between PVP and DQN baseline in MiniGrid Empty Room and Four Room and Atari Skiing tasks are provided.

Original: C E XPECTED P OLICY B EHAVIOR
Revised: C E XPECTED P OLICY B EHAVIOR

Original: In this section, we analyze the learning dynamics and provide insights on the learned policy.
Revised: In this section, we analyze the learning dynamics and provide insights on the learned policy.

Original: For simplicity, here we can define a proxy reward function:
Revised: For simplicity, here we can define a proxy reward function:

Original: [Equation]
Revised: [Equation]

Original: given by humanduring intervention
Revised: human

Original: [Equation]
Revised: [Equation]

Revised: intervention

Original: [Equation]
Revised: [Equation]

Original: Assumption 3.
Revised: Assumption 1.

Original: Assume the Q network can perfectly fit regression objective.
Revised: Assume the Q network can perfectly fit regression objective.

Original: Therefore it approximate both PVP and TD target equivalently.
Revised: Therefore it approximate both PVP and TD target equivalently.

Original: We can assume:
Revised: We can assume:

Original: [Equation][Equation]
Revised: [Equation]

Original: Finding 1 (Resembling Q Learning) .
Revised: Finding 1 (Resembling Q Learning) .

Original: Combining the PV loss with TD loss is equivalent to the traditional Q learning objective with the reward function redirected to R ( s,a )/ 2 and discount factor to γ / 2 .
Revised: Combining the PV loss with TD loss is equivalent to the traditional Q learning objective with the reward function redirected to R ( s,a )/ 2 and discount factor to γ / 2 .

Original: The convergence of our method is promised by Q learning theory.
Revised: The convergence of our method is promised by Q learning theory.

Original: Denoting the last step is T and supposing action a T leads to a terminal state, we have:
Revised: Denoting the last step is T and supposing action a T leads to a terminal state, we have:

Original: [Equation]
Revised: [Equation]

Original: Considering Assumption 3 and Eq. 15, for all steps t < T that are prior to the step T , we can easily write the bound of Q values:
Revised: Considering Assumption 1 and Eq. 10, for all steps t < T that are prior to the step T , we can easily write the bound of Q values:

Original: Theorem 3.
Revised: Theorem 2.

Original: The proxy Q value is bounded:
Revised: The proxy Q value is bounded:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: It is easy to find min a Q ( s,a ) =
Revised: It is easy to find min a Q ( s,a ) =

Original: (− 1 − γ )/ 2 ≥ − 1 .
Revised: (− 1 − γ )/ 2 ≥ − 1 .

Original: For supremum, considering the contraction of Q:
Revised: For supremum, considering the contraction of Q:

Original: [Equation]
Revised: [Equation]

Original: Finding 2 (Avoiding Value Explosion) .
Revised: Finding 2 (Avoiding Value Explosion) .

Original: The Q value is bounded so that we can eliminate the value explosion issue.
Revised: The Q value is bounded so that we can eliminate the value explosion issue.

Original: In contrast, previous work (Li et al., 2022b) uses unbounded value target so the Q value is vulnerable to overestimation and goes to infinity.
Revised: In contrast, previous work (Li et al., 2022b) uses unbounded value target so the Q value is vulnerable to overestimation and goes to infinity.

Original: Now we discuss the behavior of the final proxy Q function.
Revised: Now we discuss the behavior of the final proxy Q function.

Original: For arbitrary state action pair:
Revised: For arbitrary state action pair:

Original: [Equation]
Revised: [Equation]

Original:  t was previous human action ,
Revised: if a t was previous human action ,

Original: [Equation]
Revised: [Equation]

Original:  t was previously overwritten by human 
Revised: if a t was previously overwritten by human ,

Original: [Equation]
Revised: [Equation]

Original: otherwise 
Revised: otherwise .

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here we assume there is no attractor in state space.
Revised: Here we assume there is no attractor in state space.

Original: That is, there always exists a trajectory from current state to reach the state where human once intervened.
Revised: That is, there always exists a trajectory from current state to reach the state where human once intervened.

Original: This makes max a ′ Q ( s ′ ,a ′ )
Revised: This makes max a ′ Q ( s ′ ,a ′ )

Original: ≈ 

Revised: ≈

Original: due to contraction.
Revised: due to contraction.

Original: The optimal proxy value function sculptures such a Q value landscape that all previous human actions have highest + 1 value.
Revised: The optimal proxy value function sculptures such a Q value landscape that all previous human actions have highest + 1 value.

Original: Those actions that lead to other states where human once involved also have higher values because the + 1 value of the human-involved state will be propagated to those states.
Revised: Those actions that lead to other states where human once involved also have higher values because the + 1 value of the human-involved state will be propagated to those states.

Original: Finding 3 (Implicit Intervention Minimization) .
Revised: Finding 3 (Implicit Intervention Minimization) .

Original: Since the agent learns policy that maximizes Q function, the proxy value propagation mechanism encourages agent to be human-imitating, that is to reproduce human actions or recover to the states where human once taught what to do.
Revised: Since the agent learns policy that maximizes Q function, the proxy value propagation mechanism encourages agent to be human-imitating, that is to reproduce human actions or recover to the states where human once taught what to do.

Original: Our proxy Q value also penalizes actions that cause human intervention, which implicitly achieves intervention minimization.
Revised: Our proxy Q value also penalizes actions that cause human intervention, which implicitly achieves intervention minimization.

Original: D P ROOF OF S AFETY G UARANTEES D. T RAINING S AFETY G UARANTEE
Revised: D H UMAN I NTENT C OMPLIANCE

Original: We will derive the upper bound of the discounted probability of failure of our method, a measure oftraining safety, showing that our method can bound the training safety with the active involvement of human subject.
Revised: Here we provide a conceptual framework to describe the compliance of human intention. First, we introduce a ground-truth indicator C ∶ S × A → { 0 , 1 } of the intention violation, denoting whether the action is undesired.

Original: This proof is inspired by HACO (Li et al., 2022b) but we replace the stochastic novice policy by a deterministic policy, leading to the introduction of the notation intervention rate into the bound.
Revised: C is not revealed to learning algorithm. [Equation]

Original: In next section, we also derive  a novel testing safety guarantee.
Revised: We will derive the upper bound of the discounted occurrence of intent violation, a measure of training time human intent compliance:

Original: We first revisit the notations.
Revised: [Equation] [Equation] where P π b denotes the probability distribution of trajectories deduced by the behavior policy π b .

Original: During training, a human subject shares control with the learning agent.
Revised: During training, a human subject shares control with the learning agent.

Original: The agent’s policy is a deterministic policy µ n ( s ) , the human’s policy is a stochastic policy π h ( a ∣ s ) .
Revised: The agent’s policy is a deterministic policy µ n ( s ) , the human’s policy is a stochastic policy π h ( a ∣ s ) .

Original: The human subject intervenes I ( s,a ) =
Revised: The human subject intervenes I ( s,a ) =

Original: True under certain state and agent’s action a n .
Revised: True under certain state and agent’s action a n .

Original: The mixed behavior policy π b that produces the real actions to environment is denoted as:
Revised: The mixed behavior policy π b that produces the real actions to environment is denoted as:

Original: [Equation]
Revised: [Equation]

Original: (19) where we use Dirac delta distribution to represent the deterministic novice policy.
Revised: where we use Dirac delta distribution to represent the deterministic novice policy.

Original: The ground-truth indicator C denotes whether the action will lead to unsafe state, which is determined by the environment and is not revealed to learning algorithm:

Original: [Equation]

Original: [Equation]

Original: We measure safety with the notation cumulative discounted probability of failure as:

Original: [Equation]

Original: [Equation]

Original: which counts for the chance of entering dangerous states in current time step as well as in future trajectories deduced by the policy π .

Original: We write the expected cumulative discounted probability of failure of human subject and the behavior policy as:

Original: [Equation]

Original: Two important assumptions on the human subject are introduced.
Revised: Two important assumptions on the human subject are introduced:

Original: These two assumptions does not impose any constrain on the structure of the human expert policy.

Original: Assumption 4 (Error rate of human policy) .

Original: For all states, the step-wise probability of that the human subject produces an unsafe action is bounded by a small value ϵ < 1 :

Original: [Equation]

Original: Assumption 5 (Error rate of intervention policy) .
Revised: Assumption 2 (Error rate of human policy) .

Original: During human-AI shared control, the probability that the human subject does not intervene when the action isunsafe is bounded by a small value κ < 1 :
Revised: During human-AI shared control, the probability that the human subject produces an undesired action is bounded by a small value ϵ < 1 :

Original: [Equation]
Revised: [Equation]

Original: We utilize several useful lemmas which are used in the main theorem.

Original: Lemma 4.

Original: The cumulative probability of failure of the human S π h ( s ) is bounded for all state:

Original: [Equation]
Revised: [Equation]

Original: Proof. Following Assumption 4: [Equation] Lemma 5 (The performance difference lemma) .
Revised: Assumption 3 (Error rate of intervention policy) .

Original: The difference on the failure probability between behavior policy and human policy can be computed by:

Original: [Equation]

Original: Here the S π b and S π h represent the expected values over all possible initial states S π b / π h =

Original: E s 0 ∼ d 0 S π b / π h ( s 0 ) .

Original: P π b means the states are subject to the marginal state distribution deduced by the behavior policy π b .

Original: s ′

Original: = P( s,a ) is the next state. A π h ( s,a ) is the “cost advantage” of the human in current state action pair: A π h ( s,a ) = C ( s,a ) +
Revised: During human-AI shared control, the probability that the human subject does not intervene when the action is undesired is bounded by a small value κ < 1 :

Original: [Equation]
Revised: [Equation]

Original: π h ( s ′ )

Original: − S π h ( s ) .

Original: Lemma 5 is proposed and proved by Kakade & Langford (2002).

Original: Compared to original lemma, we replace the concept of reward by the indicator C so that the value function S π b and S π h are now the expected cumulative discounted probabilities of failure.

Original: We introduce the main theoremof this work and give the proof as follows.
Revised: We introduce the following theorem and give the proof as follows.

Original: Theorem 6 (Upper bound of training risk) .
Revised: Theorem 3 (Upper bound of intent violation) .

Original: Theexpected discounted probability of failure S π b of the behavior policy π b is bounded by the error rate of the human action ϵ , the error rate of the human intervention κ and the takeover rate ψ: [Equation] (28) wherein ψ = E s ∼ Pπb [ 1 − I ( s,a n )] denotes the intervention rate.
Revised: The discounted occurrence of intent violation S π b of the behavior policy π b is bounded by the error rate of the human action ϵ , the error rate of the human intervention κ and the intervention rate ψ = E s ∼ P

Original: Proof.

Original: We firstly decompose the advantage by splitting the behavior policy given state s :

Original: [Equation] [Equation] The second term is equal to zero according to the definition of advantage. For simplicity, denote a n = µ n ( s ) as the agent action.
Revised: πb I ( s,a n

Original: Conducting expectation over state and expanding the advantage into detailed form, we have:

Original: [Equation][Equation] [Table] [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation][Table] [Equation] c) .
Revised: [Equation]

Original: Following the Assumption 5, the term (a) is immediately bounded by κ .

Original: Following the Lemma

Original: [Equation]
Revised: [Equation]

Original: the term (b) can be written as:

Original: [Equation]

Original: [Equation]

Original: wherein ψ =

Original: E s ∼ P πb [ 1 − I ( s,a n )] denotes the overall intervention rate. The term (c) is always non-negative, so after applying the minus to term (c) the negative term will always be ≤ 0 . Aggregating the upper bounds of three terms [Equation]
Revised: Proof.

Original: we have the bound on the advantage:

Original: [Equation] [Equation] Now we put Eq.32 as well as Lemma 4 into the performance difference lemma (Lemma 5)
Revised: Consider Eq.

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Now we have proved the upper bound of the discounted probability of failure for the behavior policy in our method.

Original: Similar to the deduction of the training safety guarantee, we can also derive the testing safety guarantee supposing the Assumption 3 and Eq. 18 hold. The optimal policy will behave like:
Revised: The upper bound of S π b :

Original: [Equation]
Revised: [Equation]

Original: However, if we consider the training process together, we can write an expected novice policy as:

Original: [Equation]

Original: [Equation]

Original: ) =

Original: True denotes human subject has once intervened in state s .

Original: Note that π ∗ ( a ∣ s ) is not a outcome deterministic policy but instead a surrogate that captures the stochasticity during training and human subject.

Original: Now we can derive the testing time risk: Theorem 7 (Upper bound of testing risk) . The expected cumulative discounted probability of failure S π ∗ of the outcome policy µ ∗ ( s ) is bounded by the error rate of the human expert action ϵ , the error rate of the human expert intervention κ and human-involved state coverage ξ :
Revised: E E NVIRONMENT D ETAILS

Original: [Equation]
Revised: [Table]

Original: wherein ξ =

Original: E s ∼ P π ∗ M ( s ) measuring the coverage of human-involved states.

Original: Proof.

Original: Denote the novice policy after training is µ ∗ .

Original: The human advantage is:

Original: [Equation] [Equation] e) [Equation]
Revised: Continuous

Original: Let’s consider what the term (d) means. Suppose at state s , the deterministic action is a dangerous action C ( s,µ ∗ ( s )) = 1 and the human has not intervened at this point, namely M ( s ) = 0 . As stated in Assumption 5, this means that the human makes a mistake that has probability lower than κ .
Revised: Yes CARLA

Original: Applying the expectation over state to term (d), we now have:

Original: [Equation] Similar to the processing in Theorem 6, we have: [Equation] Here we introduce ξ =
Revised: Wheel

Original: E s ∼ P π ∗ M ( s ) as the human-involved state coverage. We reuse the policy difference lemma (Lemma 5) on novice policy and human policy. [Equation] Now we have proved the upper bound of the expected discounted probability of failure for the expected agent policy.
Revised: Bird-eye View Image

Original: E E NVIRONMENT D ETAILS

Original: [Table]
Revised: Continuous No MiniGrid Keyboard Semantic Map

Revised: Discrete

Revised: No Atari

Revised: Keyboard

Original:  To avoid the potential risks of employing human subjects in physical experiments, we benchmark different approaches in four virtual simulated environments.
Revised: RGB Image Discrete No To avoid the potential risks of employing human subjects in physical experiments, we benchmark different approaches in four virtual simulated environments.

Original: We conduct experiments on various tasks with different observation and action spaces and human input devices.
Revised: We conduct experiments on various tasks with different observation and action spaces and human input devices.

Original: Table 3 summarizes the differences.
Revised: Table 4 summarizes the differences.

Original: For continuous action space, we use two driving environments, MetaDrive Safety Benchmark (Li et al., 2022a) and CARLA Town01 environment (Dosovitskiy et al., 2017).
Revised: For continuous action space, we use two driving environments, MetaDrive Safety Benchmark (Li et al., 2022a) and CARLA Town01 environment (Dosovitskiy et al., 2017).

Original: In both tasks, the agent need to steer the target car with low-level acceleration, brake and steering and move toward destination.
Revised: In both tasks, the agent need to steer the target car with low-level acceleration, brake and steering and move toward destination.

Original: MetaDrive Safety Benchmark preserves the capacity to evaluate the safety and generalizability in unseen environments, since it uses procedural generation to synthesize an unlimited number of driving maps for the split of training and test sets, which is useful to benchmark the generalization capability of different approaches in the context of safe driving.
Revised: MetaDrive Safety Benchmark preserves the capacity to evaluate the safety and generalizability in unseen environments, since it uses procedural generation to synthesize an unlimited number of driving maps for the split of training and test sets, which is useful to benchmark the generalization capability of different approaches in the context of safe driving.

Original: We train agents in the training set, which contains 50 different scenes, and roll out the learning agents in the test set, which contains another 50 unique scenes.
Revised: We train agents in the training set, which contains 50 different scenes, and roll out the learning agents in the test set, which contains another 50 unique scenes.

Original: At each episode, the scene (road network) and the spawn location of traffic vehicles and ego vehicle are randomized.
Revised: At each episode, the scene (road network) and the spawn location of traffic vehicles and ego vehicle are randomized.

Original: We use sensory state vector in MetaDrive as the observation for agents and thus apply MLP network architecture.
Revised: We use sensory state vector in MetaDrive as the observation for agents and thus apply MLP network architecture.

Original: In CARLA, we train and test agents in the Town01 environment.
Revised: In CARLA, we train and test agents in the Town01 environment.

Original: There exist many predefined routes in the town with different spawn locations and destinations.
Revised: There exist many predefined routes in the town with different spawn locations and destinations.

Original: The route is randomized for each episode.
Revised: The route is randomized for each episode.

Original: We use the bird-eye view image in CARLA as observation and thus CNN are used as feature extractors.
Revised: We use the bird-eye view image in CARLA as observation and thus CNN are used as feature extractors.

Original: For discrete action space, we test on MiniGrid Two Room task (Chevalier-Boisvert et al., 2018) and
Revised: For discrete action space, we test on MiniGrid Two Room task (Chevalier-Boisvert et al., 2018) and

Original: Atari game Skiing task (Bellemare et al., 2013).
Revised: Atari game Skiing task (Bellemare et al., 2013).

Original: MiniGrid Two Room is a task requiring heavy exploration since the agent needs to move toward a door and open the door before reaching the destination.
Revised: MiniGrid Two Room is a task requiring heavy exploration since the agent needs to move toward a door and open the door before reaching the destination.

Original: The spawn locations, the destinations, door locations and the geometry of each room are randomized.
Revised: The spawn locations, the destinations, door locations and the geometry of each room are randomized.

Original: The observation of MiniGrid is the semantic map of agent’s local neighborhood.
Revised: The observation of MiniGrid is the semantic map of agent’s local neighborhood.

Original: MiniGrid tasks only support using keyboard as input device.
Revised: MiniGrid tasks only support using keyboard as input device.

Original: Only in the MiniGrid task, we render the agent’s action in the environment so that the human can decide whether to take over or return back based on both current state and agent’s action.
Revised: Only in the MiniGrid task, we render the agent’s action in the environment so that the human can decide whether to take over or return back based on both current state and agent’s action.

Original: But this is not feasible in other tasks since other tasks require real-time response from humans and there is not enough time for humans to observe agent’s actions even if we plot those actions in visualization interface.
Revised: But this is not feasible in other tasks since other tasks require real-time response from humans and there is not enough time for humans to observe agent’s actions even if we plot those actions in visualization interface.

Original: The Atari game is difficult since the agent needs to learn to output meaningful action based on the
Revised: The Atari game is difficult since the agent needs to learn to output meaningful action based on the

Original: RGB observation.
Revised: RGB observation.

Original: We use the default setting provided by the Gym Atari environment.
Revised: We use the default setting provided by the Gym Atari environment.

Original: We experiment on Atari Skiing game.
Revised: We experiment on Atari Skiing game.

Original: In preliminary experiments, we tried a few other Atari games.
Revised: In preliminary experiments, we tried a few other Atari games.

Original: We find that human-in-the-loop method is not applicable to many games in Atari that have long horizons or are hard to play by human experts.
Revised: We find that human-in-the-loop method is not applicable to many games in Atari that have long horizons or are hard to play by human experts.

Original: For example, in the Breakout where the player must knock down as many bricks as possible by using the paddle below to ricochet the ball against the bricks and eliminate them, a full episode requires more than 120 seconds (can be much longer to reach a higher score) and the human expert has to attentively focus on playing the game.
Revised: For example, in the Breakout where the player must knock down as many bricks as possible by using the paddle below to ricochet the ball against the bricks and eliminate them, a full episode requires more than 120 seconds (can be much longer to reach a higher score) and the human expert has to attentively focus on playing the game.

Original: It is quite common for the human expert to miscalculate the trajectory and fail to rescue the ball from falling down.
Revised: It is quite common for the human expert to miscalculate the trajectory and fail to rescue the ball from falling down.

Original: Another example is the Enduro, where the player controls a super fast racing car to run on an infinite long track.
Revised: Another example is the Enduro, where the player controls a super fast racing car to run on an infinite long track.

Original: Experts cannot always give the optimal intervention in such a fast moving scene.
Revised: Experts cannot always give the optimal intervention in such a fast moving scene.

Original: In these environments, a pure RL agent will usually perform better even than the human experts.
Revised: In these environments, a pure RL agent will usually perform better even than the human experts.

Original: Besides as we discussed in the Limitation, representation learning is also a major challenge.
Revised: Besides as we discussed in the Limitation, representation learning is also a major challenge.

Original: It takes a long time for the convolutional neural network to learn good representations from the RGB frames.
Revised: It takes a long time for the convolutional neural network to learn good representations from the RGB frames.

Original: Therefore, the human-in-the-loop methods with active human involvement are not satisfactory in Atari games.
Revised: Therefore, the human-in-the-loop methods with active human involvement are not satisfactory in Atari games.

Original: F E XTRA E XPERIMENTAL R ESULTS F.I MPACT OF C ONTROL D EVICES
Revised: F E XTRA E XPERIMENTAL R ESULTS F. C OMPARING PVP AND HACO

Original: [Table]

Original: Table 4 presents the experiment results with different input devices in MetaDrive benchmark.

Original: In both settings, the agents trained by PVP outperform those by HACO, showing the generalizability of PVP on different input devices

Original: [Table]

Original: [Figure]

Original: C OMPARING PVP WITH HACO

Original: In Table 4, we observe that HACO (Li et al., 2022b) has performance discrepancy with different input devices.

Original: When using Gamepad, human subjects tend to push and pull the stick to the limits, producing extreme values.

Original: As we will discuss later, extreme actions are particularly harmful to

Original: HACO.

Original: When using keyboard, the human subjects press arrow keys to indicate increasing/decreasing current steering/acceleration values for an increment.

Original: Therefore there will be fewer extreme values happening than using a Gamepad, which explains why the baseline HACO performs better with the keyboard compared to Gamepad.

Original: [Figure]
Revised: [Figure]

Original: To explain why HACO is vulnerable toextreme action values, we compare the proxy Q values in our method and in HACO baseline in Fig. 10 and find that HACO has much larger magnitude in its proxy values compared toPVP. This is because HACO updates the state-action values based on CQL objective (Kumar et al., 2020): min Q ( s,a n ) − Q ( s,a h ) . Since extreme values happen frequently when human intervenes through Gamepad, the proxy values in those human actions are reinforced without bound, making the novice policy rapidly learn those actions.
Revised: Randomness in HACO’s policy is a reason leading to the suboptimal performance compared to

Original: In contrast, the proxy values learned via PVP are much more moderate. The novice Q has distinct negative value. In contrast, the values of behavior actions, the actions that satisfy human, have positive values. This result reveals the problem of unbounded values in the previous method.
Revised: PVP.

Original: Our method instead resolves this issue and has bounded proxy values, leading to stable training.

Original: [Figure]

Original: Another possible reason to explain the performance drop is the randomness in HACO’s policy.

Original: We run the agents trained from HACO and PVP in CARLA task in a straight road in CARLA town and plot their actuating signals in Fig.
Revised: We run the agents trained from HACO and PVP in CARLA task in a straight road in CARLA town and plot their actuating signals in Fig.

Original: In this task, the steering should be always close to zero but it turn out that, as the training iterations increase, the HACO agents gradually demonstrate unstable steering.
Revised: In this task, the steering should be always close to zero but it turn out that, as the training iterations increase, the HACO agents gradually demonstrate unstable steering.

Original: In human-AI shared control, such unstable behaviors force human subjects to involve frequently.
Revised: In human-AI shared control, such unstable behaviors force human subjects to involve frequently.

Original: In contrast, the PVP policies learn a much better solution in lane keeping.
Revised: In contrast, the PVP policies learn a much better solution in lane keeping.

Original: In Appendix F.3, we further present the visualization of the trajectories generated by human-robot shared control.
Revised: In Appendix F.2, we further present the visualization of the trajectories generated by human-robot shared control.

Original: The actions proposed by PVP policy are smoother than those by HACO.
Revised: The actions proposed by PVP policy are smoother than those by HACO.

Original: These results explain the performance of PVP in CARLA task and is aligned with the behavior shown in the supplementary video.
Revised: These results explain the performance of PVP in CARLA task and is aligned with the behavior shown in the supplementary video [Figure]

Original:  V ISUALIZATION OF H UMAN - ROBOT S HARED C ONTROL
Revised: F. V ISUALIZ A ATION OF H UMAN - ROBOT S HARED C ONTROL

Original: HACO @ 10K steps
Revised: HACO @ 10K steps

Original: HACO @ 20K steps HACO @ 30K steps
Revised: HACO @ 20K steps HACO @ 30K steps

Original: HACO @ 40K steps
Revised: HACO @ 40K steps

Original: PVP @ 10K steps
Revised: PVP @ 10K steps

Original: PVP @ 20K steps PVP @ 30K steps
Revised: PVP @ 20K steps PVP @ 30K steps

Original: PVP @ 40K steps
Revised: PVP @ 40K steps

Original: In Fig.
Revised: In Fig.

Original: 12, we present the visualization of the trajectories during human-robot shared control.
Revised: 11, we present the visualization of the trajectories during human-robot shared control.

Original: Comparing the visualization of HACO and PVP, we find that PVP generates smoother trajectories.
Revised: Comparing the visualization of HACO and PVP, we find that PVP generates smoother trajectories.

Original: The credit belongs to the deterministic policy we used.
Revised: The credit belongs to the deterministic policy we used.

Original: Stable and smooth agent actions greatly improve human subjects’ experience and relieve their stress during human-robot shared control.
Revised: Stable and smooth agent actions greatly improve human subjects’ experience and relieve their stress during human-robot shared control.

Original: We can also find that as the training goes, PVP requires less human involvement.F. A BLATION S TUDY [Table] [Table]
Revised: We can also find that as the training goes, PVP requires less human involvement.

Original: We conduct an ablation study to show the importance of TD learning and balanced buffers.

Original: As shown in Table 5, disabling TD learning via setting J TD ( Q ) = 0 greatly damages the performance of PVP.

Original: This suggests that our Principle 2 is critical to the success of PVP.

Original: In Table 6, we find that disabling balanced buffers makes the training unstable and leads to poor performance.

Original: Table 5 also presents the experiment result when we train PVP with environmental reward.

Original: We find that reward has no significant impact on the learning performance

Original: [Table]

Original: F. E XTRA R ESULTS 
Revised: F. E XTRA R ESULTS IN M INI G RID

Original: [Figure] M INI G

Original: [Figure]
Revised: [Figure]

Original: In Fig.
Revised: In Fig.

Original: [Figure]
Revised: [Figure]

Original: we present the extra results in two additional MiniGrid environments.
Revised: we present the extra results in two additional MiniGrid environments.

Original: PVP achieves superior performance compared to RL baseline.
Revised: PVP achieves superior performance compared to RL baseline.

Original: Note that we use a CNN without recurrent module as the feature extractor.
Revised: Note that we use a CNN without recurrent module as the feature extractor.

Original: The performance of PVP can be further improved if we utilize the neural architecture with memory capacity.
Revised: The performance of PVP can be further improved if we utilize the neural architecture with memory capacity.

Original: G H YPER - PARAMETERS
Revised: G H YPER - PARAMETERS

Original: In MetaDrive safety benchmark (Li et al., 2022a) task, the observation is a state vector.
Revised: In MetaDrive safety benchmark (Li et al., 2022a) task, the observation is a state vector.

Original: There exists a split of training and test environments in MetaDrive.
Revised: There exists a split of training and test environments in MetaDrive.

Original: We present the result of learned agent performing in test environment.
Revised: We present the result of learned agent performing in test environment.

Original: In CARLA (Dosovitskiy et al., 2017), the observation is the bird-eye view image in [ 84 , 84 , 5 ] shape, where 5 is the number of semantic channels.
Revised: In CARLA (Dosovitskiy et al., 2017), the observation is the bird-eye view image in [ 84 , 84 , 5 ] shape, where 5 is the number of semantic channels.

Original: We train and evaluate the agents in the same
Revised: We train and evaluate the agents in the same

Original: NoCrashTown01 environment.
Revised: NoCrashTown01 environment.

Original: In both driving tasks, the agent needs to steer the target car with low-level acceleration, brake and steering and move toward destination and thus the action space is a two dimensional space.
Revised: In both driving tasks, the agent needs to steer the target car with low-level acceleration, brake and steering and move toward destination and thus the action space is a two dimensional space.

Original: In MiniGrid tasks (Chevalier-Boisvert et al., 2018)
Revised: In MiniGrid tasks (Chevalier-Boisvert et al., 2018)

Original: MiniGrid-Empty-Random-6x6-v0 (Empty Room),
Revised: MiniGrid-Empty-Random-6x6-v0 (Empty Room),

Original: MiniGrid-MultiRoom-N2-S4-v0 (Two Room) and MiniGrid-MultiRoom-N4-S5-v0 (Four Room), the observation is the top-down view semantic map in shape [ 7 , 7 , 3 ] .
Revised: MiniGrid-MultiRoom-N2-S4-v0 (Two Room) and MiniGrid-MultiRoom-N4-S5-v0 (Four Room), the observation is the top-down view semantic map in shape [ 7 , 7 , 3 ] .

Original: In Atari game Skiing task (Bellemare et al., 2013), the observation is originally [210, 160] and we resize and preprocess the images following (Mnih et al., 2015).
Revised: In Atari game Skiing task (Bellemare et al., 2013), the observation is originally [210, 160] and we resize and preprocess the images following (Mnih et al., 2015).

Original: We also stack 4 consecutive frames so the input to neural network is in shape [84, 84, 4].
Revised: We also stack 4 consecutive frames so the input to neural network is in shape [84, 84, 4].

Original: For CARLA and Atari tasks, since the input image has the same size of [84, 84] pixels, we use the same 5-layers CNN architecture with [16, 32, 64, 128, 256] filters in each layers.
Revised: For CARLA and Atari tasks, since the input image has the same size of [84, 84] pixels, we use the same 5-layers CNN architecture with [16, 32, 64, 128, 256] filters in each layers.

Original: The corresponding kernel-size is [[4, 4], [3, 3], [3, 3], [3, 3], [4, 4]], and strides [3, 2, 2, 2, 4].
Revised: The corresponding kernel-size is [[4, 4], [3, 3], [3, 3], [3, 3], [4, 4]], and strides [3, 2, 2, 2, 4].

Original: We use ReLU as activation functions between each layer.
Revised: We use ReLU as activation functions between each layer.

Original: For MiniGrid tasks, we use 3-layers CNN architecture with [16, 16, 32] filters in each layer.
Revised: For MiniGrid tasks, we use 3-layers CNN architecture with [16, 16, 32] filters in each layer.

Original: All three layers have kernel-size 2 and there is a max-pooling layer between the first two layers.
Revised: All three layers have kernel-size 2 and there is a max-pooling layer between the first two layers.

Original: We use ReLU as activation functions between each layer.
Revised: We use ReLU as activation functions between each layer.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Value 0.