From: ICLR.cc/2023/Conference/Paper1917/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper1917/Authors
title: Response
comment: Sorry for the late response.


> I see that the numbers for SAC, PPO, SAC-Lag, PPO-Lag, and CPO are identical. So is it that the numbers in Table 1 of the HACO paper for the RL algorithms are for MetaDrive-Keyboard, but the numbers in the same table for HACO are for MetaDrive-Wheel?

For RL baselines, such as SAC, PPO, SAC-Lag, PPO-Lag, CPO, they do not use demonstration provided by human expert. Therefore we can copy the result from HACO papers since whether to use Keyboard of Wheel makes no differences for these baselines.

---


> So we see that the numbers for HACO in the original paper are still much better than the numbers in your paper, even on MetaDrive-Wheel. In fact, if we use the numbers in the original HACO paper, we see that HACO outperforms PVP on all metrics using less data, while your paper says the exact opposite. 

As we say we report the results generated by our human subjects. We wish to maintain the consistency of the human subjects involved in the experiment.

From: ICLR.cc/2023/Conference/Paper1917/Reviewer_hHb7
title: Thanks for the response
comment: Sorry for the late response.

Thanks for the updates to the theory, I think it is much simpler and easier to understand now.

> First, we want to clarify that the HACO baseline shown in Table 1 is experimented in MetaDrive-Keyboard environment while the original work of HACO is using MetaDrive-Wheel. Therefore the setting is not the same.

Comparing the numbers in your Table 1 (MetaDrive-Keyboard) with the numbers in Table 1 (MetaDrive) in the HACO paper, I see that the numbers for SAC, PPO, SAC-Lag, PPO-Lag, and CPO are identical. So is it that the numbers in Table 1 of the HACO paper for the RL algorithms are for MetaDrive-Keyboard, but the numbers in the same table for HACO are for MetaDrive-Wheel?

> We already compared PVP with HACO in MetaDrive-Wheel environment and the discrepancy of HACO is not large in MetaDrive-Wheel.

Is it not? In MetaDrive-Wheel, your paper says:

PVP: With 40K training data and 10.3K human data, at test time PVP achieves 336.657 episodic return, 1.543 safety cost, and 0.808 success rate.
HACO: With 40K training data and 21.2K human data, at test time HACO achieves 250.039 episodic return, 1.453 safety cost, and 0.355 success rate.

However, the original HACO paper says:

HACO: With 30K training data and 8.3K human data, at test time HACO achieves 349.25 episodic return, 0.79 safety violations, and 0.83 success rate.

So we see that the numbers for HACO in the original paper are still much better than the numbers in your paper, even on MetaDrive-Wheel. In fact, if we use the numbers in the original HACO paper, we see that HACO outperforms PVP on all metrics using _less_ data, while your paper says the exact opposite. (And the difference is substantial.)

From: ICLR.cc/2023/Conference/Paper1917/Reviewer_RMcE
title: Response to authors
comment: Sorry for the late response, but i now managed to read up on your changes. Thank you for the extensive feedback. The introduced changes improve the paper substantially and therefore I'm willing to increase my score. Of course, my novelty concerns remain and additional results would be welcome.

From: ICLR.cc/2023/Conference/Paper1917/Authors
title: Update
comment: We update the new experiment results of the human-in-the-loop baselines in the discrete tasks. You can see that our proposed method achieve best efficiency and performance. Many other human-in-the-loop baselines are running.

We are very disappointed by the fact that most reviewers keep irresponsive during the discussion period. Thanks.





Table: Comparing baselines that involve human data in MiniGrid task

|            | Human Data Usage | Total Data Usage | Success Rate |
|------------|------------------|------------------|--------------|
| BC         | 100              | -                | 0.30         |
| BC         | 200              | -                | 0.41         |
| BC         | 500              | -                | 0.90         |
| IWR        | 265 (0.164)      | 1619             | 0.96         |
| PVP (Ours) | 128.84 (0.085)   | 1504.71          | 1.00         |

From: ICLR.cc/2023/Conference/Paper1917/Authors
title: Thank all reviewers and we revised the submission
comment: We thank all reviewers for the insightful reviews. We revise the paper to include suggestions proposed by reviewers. The major changes, which are marked by red color in the paper, are summarized as follows:

1. We simplify the theoretical analysis according to the suggestion of Reviewer hHb7 and uDKA.
2. Suggested by Reviewer RMcE, we move the detailed comparison of PVP and human-in-the-loop baseline and ablation studies from the appendix to the experiment section of the main paper.
3. We conduct a new ablation study of incorporating PVP with Soft Actor-critic, to investigate the impact of stochastic novice policy. The experiment is conducted by real human. The result suggests that the deterministic novice policy can lead to better training efficiency. This experiment addresses the concern raised by Reviewer uDKA and RMcE.

In the meantime, we are conducting baselines experiments, namely BC/GAIL/IWR/HG-DAgger in other tasks with our human subjects. We will update the submission once results are ready.

From: ICLR.cc/2023/Conference/Paper1917/Authors
title: Response to Reviewer RMcE (2/2) 
comment: **Q3**: Core findings comparing PVP with existing active human involvement methods should be in the main body of the paper. 

**A3**: We revise the paper and move the detailed comparison of PVP with HACO to Section 5.3. We discuss the impact of different input devices and the reason why HACO is outperformed by PVP.

---

**Q4**: Goal 2 is not intuitive. Why do we assume the agent should reach human-involved states? The reasoning is mostly based on how the algorithm works but not the other way around.

**A4**: We revise the paper to reduce confusion. Our intuition is that human intervention and demonstration are the only source of supervision that we can learn from. TD learning can propagate the information of those desired actions provided by humans to other states. In revision, we move the ablation study to the main body and Table 1 shows that TD learning has a huge impact on the learning performance. 

---

**Q5**: Figure 5 shows a minor amount of interventions across the board. However, in the first iterations, interventions are expected to be dominating, because the initial (random?) policy is likely to behave very badly. In case the initial policy is more informed, this needs to be clarified.

**A5**: Figure 5 in original submission presents the intervention frequency, the average number of the segments of trajectory provided by human subjects in each episode. It is not equivalent to the intervention rate which is the ratio of human-involved steps over total time steps. 

To better present our findings, we revise Figure 5 to include both the intervention frequency and the intervention rate. We can find that compared to HACO, PVP can effectively reduce the intervention rate without causing high intervention frequency. This shows that both the temporal duration of human involvement and the frequency of human involvement are reduced.

From: ICLR.cc/2023/Conference/Paper1917/Authors
title: Response to Reviewer RMcE (1/2)
comment: Thanks for your review. We provide the following responses to alleviate your concerns:

---

**Q1**: The positioning of this work. This work should be in “learning from demonstration” and “learning from advice”. Many works use the same principle of updating Q values, such as TAMER, works of Najar et al. and HACO.

**A1**: Thanks for referring to insightful works, we revise the paper to include the discussion. As suggested by Reviewer uDKA, we adopt an understudied setting **active human involvement**, which is different from learning from batched demonstrations or post-hoc trajectory comparisons. A human subject is involved and monitoring the free exploration of the agent.

In conventional learning from demonstration approaches, the training happens after the data is collected and human subjects are out of the loop. Imitation learning, offline RL method and inverse RL methods  fall into this domain. Our method also learns from the demonstration, but the demonstrations are generated in an online manner. The states prior to the starting state of human demonstrations are deduced by the novice policy. This greatly improves the efficiency since the distributional shifts are reduced. Besides, human intervention also serves as a good supervision since it indicates the action is undesired.

Many works focus on active human involvement. Apart from many human-in-the-loop methods we discussed in the original paper, there are many works that utilize the idea of learning from advice. 
The TAMER framework is a classic instance. It learns policy from human-provided evaluative feedback (human-generated reward). The Task-Instruction-Contingency-Shaping (TISC) method proposed in (Najar et al. 2020) also uses evaluative feedback. Evaluative feedback is a boolean criticizing correct/wrong. This is similar to the intervention in our framework. There are several differences between PVP and TICS:

1. Reward function exists to ensure the autonomy of the learning process.
2. Humans are involved in different forms: (1) evaluative feedback and (2) instructions. 
3. In TICS, humans provide high-level instructions, e.g. pointing to the left/right, while in PVP humans provide intervention and low-level demonstrations.
4. TICS has a task model improving policy directly based on the evaluative feedback. The difference is that PVP uses a proxy value to indirectly encourage agents to follow human preference, while TICS directly modifies the policy action distribution.

Compared to learning from advice methods, PVP requires human subjects to provide low-level demonstrations when intervening and does not ask for humans’ evaluative rewards.



(Najar et al. 2020) Najar, Anis, Olivier Sigaud, and Mohamed Chetouani. "Interactively shaping robot behavior with unlabeled human instructions." Autonomous Agents and Multi-Agent Systems 34.2 (2020): 1-35.

(Knox et al. 2008) Knox, W. Bradley, and Peter Stone. "Tamer: Training an agent manually via evaluative reinforcement." 2008 7th IEEE international conference on development and learning. IEEE, 2008.

---

**Q2**: It is not clear if stochastic policy affects the human subjects.

**A2**: Please note that in Appendix F.2 of original submission, we present the trajectories of human-robot shared control. Comparing the visualization of HACO and PVP, we find that PVP generates smoother trajectories. This is a result of the deterministic policy we used. Stable and smooth agent actions greatly improve human subjects' experience and relieve their stress during human-robot shared control.

To better illustrate this, we conduct a new ablation study. We implement the core innovations of PVP in the Soft Actor-critic framework (called PVP w/ SAC) so that the novice policy is now a stochastic policy and run the experiment in CARLA. In SAC, there is one extra objective to maximize the entropy of the action distributions. We can better validate the influence of the deterministic policy we propose to use in PVP. 

The human subjects report that the novice agents in PVP w/ SAC oscillate frequently, making them hard to respond when the agents suddenly drive toward the sideroad. The qualitative evaluation of the learned agents are shown in the following table. We can find that PVP w/ SAC performs worse compared to the deterministic version of PVP.

| Method    | Data Usage | Episodic Return | Route Completion | Success Rate |
|-----------|------------|-----------------|------------------|--------------|
| PVP + SAC | 18K        | 200.18          | 0.39             | 0.13         |
| PVP + TD3 | 18K        | 449.65          | 0.76             | 0.49         |

From: ICLR.cc/2023/Conference/Paper1917/Authors
title: Response to Reviewer 6qMp
comment: Thank you for the review and acknowledgement. Please find detailed responses to your concerns below:

---

**Q1**: The motivation is not clear: in the introduction you try to learn human intent but then focus on ensuring safety.

**A1**: Our motivation is to learn a policy that satisfies human preference. Human intent is usually multi-objective and safety is one of the human intentions. In this work, safety, efficiency and satisfactory behaviors are all desired. Among them, the task performance and safety violation are two intuitive and useful measurements of human intent.

To avoid confusion and make the result more general, we change the title of Sec 4.3 from “Safety Guarantee” to “Human Intent Compliance” and change the safety violation indicator to the intent violation indicator. This does not affect much in math but can better reflect our motivation.


---

**Q2**: Why should the value of the action be lower than all other actions? It is possible that all other actions were equally unsafe. I would like the authors to clarify why this is a valid argument to make?

**A2**: As suggested by Reviewer uDKA, if we don’t suppress the values of the agent’s actions that are intervened by humans, there would be a risk of spurious positive reward cycles and lead to policies that intentionally behave in an undesired manner to elicit human intervention. Though it is possible that “other actions are equally unsafe”, the information we can get from the fact that the human subject intervenes at this action is that the agent’s action is not desired and we should optimize the policy to avoid these undesired actions. As training goes, even those “equally unsafe actions” will be explored by the agent and be suppressed if human intervenes.

---

**Q3**: For those human-involved states, these states are more “unsafe” than other states. Why do we transit back to those states?

**A3**: We revise the paper to moderate confusion. Our intuition is that human intervention and demonstration are the only source of supervision that we can learn from. TD learning can propagate the information of those desired actions provided by humans to other states. In revision, we move the ablation study to Table 1 in the main paper and show that TD learning has a huge impact on the learning performance. 

As for the safety issue, since we assume human subjects have a bounded error rate when making intervention (Assumption 2 in Appendix D), human subjects will continuously improve the quality of the actions and even avoid those near-accidental states. Besides, as training goes the agent learns to avoid the undesired actions and thus those states become “safer”.


---

**Q4**: If this is in fact a safety-via-human-advice style of work (instead of preference learning), does the argument that reward specification is a hard problem still apply? Can’t the tasks we are using be solved by reward functions? That is, can you describe to what extent, in the domains you have considered for your experiments, it is hard to specify safety violations explicitly [to justify an HiL approach].

**A4**: First, we are not safety-via-human-advice. We want to learn human preference. Safety is only one of the preferences of humans. Therefore reward specification is still hard in our setting since human preference is usually multi-objective.

Even though it seems that reward specification is not a problem when we only consider optimizing safety, our proposed method addresses another important problem that exists in current safe RL literature: the training safety. A safety reward specification cannot guarantee the training time safety because optimal actions are gained through safety-prone explorations, which is also reflected in much poorer results in Safe RL experiments. PVP achieves much less training-time safety violation due to active human involvement.

From: ICLR.cc/2023/Conference/Paper1917/Authors
title: Response to Reviewer uDKA (2/2) 
comment: 
**Q4**: There would seem to be a risk that assigning reward to human’s actions could lead to spurious positive reward cycles, or could even lead to policies that intentionally get into difficult parts of the state space to elicit human intervention.

**A4**: We already considered this issue. First, when formulating the problem in Section 3, we say one of the goals of our design is to minimize human involvement. Second, in Goal 1, we say we want to suppress undesired actions and we propose to assign -1 Q value to those actions that are intervened by human subjects. If the agent learns to deliberately cause dangers that require human intervention, then those actions will be assigned -1 value and never be chosen (unless human intervenes by mistake and provides human action same to agent action).


---

**Q5**: Should address the possibility that, in tasks with high action frequency, humans may intervene after a single mistaken action, even if later actions were correct.

**A5**: Our method can fit this situation since, during human involvement, if agent actions are correct and human actions are similar to them, the state-action value will be assigned to 0 (optimizing to +1 and -1 simultaneously using regression). Therefore the agent's actions will not be penalized.

---

**Q6**: Explain how the baseline RL algorithms were tuned. In figure 4, the sub-figure for DQN over 10 million steps suggests that performance on the Atari task actually drops, suggesting a problem with the choice of parameters. How does this result compare to the state-of-the-art for this Atari game?

**A6**: Please note that, to ensure fairness, we always use the same hyper-parameters for both the PVP RL counterpart and the RL algorithm. The best reported result of DQN in the Skiing game is -13062.3 (https://arxiv.org/pdf/1511.06581.pdf). Such a result is achieved after millions of steps while our approach can achieve comparable performance in 20K steps. Both our method and vanilla DQN can’t outperform the SOTA result which achieves ~-7500 reward. We don’t aim at surpassing SOTA. Considering the fact that we use vanilla DQN as backbone and Atari is particularly challenging in representation learning, especially for our human-in-the-loop methods, our result proves that it is possible to use HL method to achieve comparable performance to its RL counterpart with much less data usage.

From: ICLR.cc/2023/Conference/Paper1917/Authors
title: Response to Reviewer uDKA (1/2)
comment: Thank you for your review. We are happy to see you acknowledge the importance of the research and the simplicity and effectiveness of our method. In particular, we appreciate that you identify the difference between our method and existing literature on offline learning and post-hoc trajectory comparisons. Please find detailed responses to your concerns below:

---

**Q1**: Theorem 2 is weak.

**A1**: Thanks for the insightful questions. We revise the theory part and remove the test safety guarantee since it does not contain enough insight. Currently, the theory part is simplified to present a conceptual framework to describe the compliance of human intention (and safety can be included in the human intention). If the human subjects can provide high quality demonstration and intervention, and the learning algorithm is capable of reducing the intervention rate, then the human-AI system will be compliant to the underlying human intention, and in safety-critical tasks ensure safety.


---

**Q2**: Lack of evaluation of human’s experience.

**A2**: Indeed, the experience of human subjects is hard to measure. In this work we use safety violations and human data usage to measure the cognitive cost of human subjects. 

Please note that in Appendix F.2 of original submission, we present the trajectories of human-robot shared control. Comparing the visualization of HACO and PVP, we find that PVP generates smoother trajectories. This is a result of the deterministic policy we used. Stable and smooth agent actions greatly improve human subjects' experience and relieve their stress during human-robot shared control.

To better illustrate this, we conduct a new ablation study. We implement the core innovations of PVP in the Soft Actor-critic framework (called PVP w/ SAC) so that the novice policy is now a stochastic policy and run the experiment in CARLA. In SAC, there is one extra objective to maximize the entropy of the action distributions. We can better validate the influence of the deterministic policy we propose to use in PVP. 

The human subjects report that the novice agents in PVP w/ SAC oscillate frequently, making them hard to respond when the agents suddenly drive toward the sideroad. The qualitative evaluation of the learned agents are shown in the following table. We can find that PVP w/ SAC performs worse compared to the deterministic version of PVP.

| Method    | Data Usage | Episodic Return | Route Completion | Success Rate |
|-----------|------------|-----------------|------------------|--------------|
| PVP + SAC | 18K        | 200.18          | 0.39             | 0.13         |
| PVP + TD3 | 18K        | 449.65          | 0.76             | 0.49         |

We revise the paper to remove the “well-being” statements.

---

**Q3**: Human teaching behavior may be different compared to the expected behaviors. We claim in section 3 that the lack of human intervention indicates that the learned policy is behaving correctly, while humans only intervene when behavior is incorrect. This might not be true in other cases. The authors could devote more space to discussing the difficulty of interpreting human behavior, and the extent to which human teaching behavior might change in environments other than the ones (primarily autonomous driving) considered in their experiments.

**A3**: Indeed, interpreting human behavior is hard. We don’t dive deeply to constrain the particular behavior of human subjects. As we stated in the theory part of the paper in Sec. 4.3, we introduce two assumptions on human behavior: their actions and their interventions are mostly correct enough to reflect a ground-truth indicator of human intention: C(s, a) = 0 if human intent is not violated and 1 otherwise.

In the conceptual framework we proposed, your example that humans wish to intervene when the agent is close to the goal can be interpreted as follows: human intent is to let agents achieve goals as fast as possible when the agents are near the goal. 

If we are expecting the human subjects to provide demonstrations in the beginning of episodes and the human subjects behave in your proposed manner, then this is violating our Assumption 2 on the “intervention error rate” so this will lead to a poor novice policy as suggested by the theorem in Sec. 4.3.

From: ICLR.cc/2023/Conference/Paper1917/Authors
title: Response to Reviewer hHb7
comment: Thank you for your detailed review. Please find detailed responses to your concerns below:

---

**Q1**: The theorem is overcomplicated. If you change Assumption 2 by replacing the expectation over the behavior policy to the novice policy, then Theorem 1 and 2 will be simplified but trivial.

**A1**: We revise the theory part according to your suggestion to make it more concise.

First, we apply your suggestion and modify Assumption 2 to be the expectation over novice policy. 

Second, we adopt your idea and greatly simplify the proof. However, in your proof the term $P_{I}(s) = \mathbb E_{a\sim \pi_n} I(s, a)$ is discarded while we keep it since it is the intervention rate $\psi$ after applying the expectation over states.

Third, we rephrase the term $C(s, a)$ to be a boolean indicator of the intention violation, instead of the safety violation. This does not affect the math and makes the paper more consistent.

Now, the upper bound of the human intention violation during training can be written as:

$  S_{\pi_b} \le \cfrac{1}{1 - \gamma}(\kappa + \epsilon \psi)  $

This simple bound frames a conceptual guarantee on the compliance of human intention. If the human subjects can provide high quality demonstration and intervention, and the learning algorithm is capable of reducing the intervention rate, then the human-AI system will be compliant to the underlying human intention. 

Since this result is straightforward, we largely simplify Section 4.3 in the main paper and replace the spare space by important comparison with baselines in experiment in Section 5.3.


---


**Q2**: The baselines for comparison should primarily be other approaches using human data.

**A2**: We are conducting new experiments of GAIL, IWR and HG-DAgger in various tasks. Please stay tuned.

---

**Q3**: Discrepancy in the HACO baseline. The RL results for MetaDrive-Keyboard are mostly copied from the HACO paper, suggesting that **the setting is the same** as in the HACO paper. Numerical discrepancies with the HACO paper suggest **poorly tuned baselines**.

**A3**: First, we want to clarify that the HACO baseline shown in Table 1 is experimented in MetaDrive-Keyboard environment while the original work of HACO is using MetaDrive-Wheel. Therefore the setting is not the same. 

We already compared PVP with HACO in MetaDrive-Wheel environment and the discrepancy of HACO is not large in MetaDrive-Wheel. The detailed discussion was originally in the Appendix. We move the performance of PVP and HACO with multiple input devices and the comprehensive discussion on the dropped performance of HACO to Section 5.3 of the revised paper.

Second, as for the hyper-parameter tuning, we don’t change the hyperparameter of HACO. The discrepancy are stemmed from the difference in human subjects. We want to ensure a fair comparison so we have to present the reproducing results of human-in-the-loop baselines with our human subjects.

---

**Q4**: The statement seems false: “We introduce two important assumptions on the human subject without imposing any constraint on the human policy”.

**A4**: We remove “without constraint” in this sentence. We are saying there is no assumption on the particular behaviors of the human policy and we only roughly describe its properties by the error rates on actions and interventions.

From: ICLR.cc/2023/Conference/Paper1917/Reviewer_hHb7
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper aims to learn policies in the reward-free setting, where the task is instead defined through human feedback. The core idea is to have humans intervene when the agent takes bad actions, and then say that the action chosen by the human has a Q-value of +1 while the action chosen by the agent has a Q-value of -1. These are then propagated back by the standard TD learning loss to learn a full policy. This is called Proxy Value Propagation (PVP).

The authors prove that if we assume that the human (1) very rarely takes unsafe actions, and (2) very rarely fails to intervene when the agent takes an unsafe action, then we have an upper bound on the expected discounted number of safety violations.

Experimental results on the MetaDrive-Keyboard show that PVP outperforms a variety of baselines. Results on several other environments show that it outperforms various RL algorithms (which have access to the environment reward) and a previous method HACO.
strength_and_weaknesses: Strengths:
1. The method is conceptually very simple and (presumably) easy to implement.
2. The experimental results on MetaDrive-Keyboard appear to be quite strong.

Weaknesses:
1. The theoretical safety bounds seem quite trivial (and weaker than they should be), see below.
2. The results for HACO are very low, suggesting that the baselines were not tuned appropriately.
3. For the remaining environments, the most relevant baselines for a method with active human involvement would be other methods that use human data, but in most cases the method is instead compared to RL. The only human-in-the-loop baseline used consistently is HACO.


**Theoretic safety guarantees**

These guarantees do not seem particularly compelling to me: they basically amount to saying “if the failure probability on each individual timestep is small, then the failure probability over the entire trajectory can also be bounded”. In addition, the proof method that the authors use seems overly complicated.

Part of the complication of the proof is because the authors use an assumption on the intervention policy conditioned on the _behavior policy_ $\pi_b$ rather than the _agent policy_ $\pi_n$, as would be more natural. (Conditioning on the behavior policy amounts to saying “after the human might have intervened to take a safe action, if the action is unsafe, what is the probability that the human intervened”; this is much less natural than saying “what is the probability that the human would intervene if the agent produced an unsafe action”.)

Suppose we change Assumption 2 to use $\pi_n$ instead of $\pi_b$, thus saying $E_{s,a ~ P_{\pi_n}} (1 − I(s, a))C(s, a) \leq \kappa$.

Then we have the following simple proof of an upper bound. Let $P_I(s) = \int_{a’ \in A} I(s, a’) \pi_n(a’ \mid s) da’$ be the probability that the human intervenes at state $s$. Then we have:

$E_{a ~ \pi_b} C(s, a) = \int_{a \in A} [ (1 − I(s, a)) \pi_n(a \mid s) + P_I(s) \pi_h(a \mid s) ] C(s, a) da$

Since $P_I(s) \leq 1$, we can conclude:

$E_{a ~ \pi_b} C(s, a) \leq [ \int_{a \in A} (1 − I(s, a)) \pi_n(a \mid s) C(s, a) da ] + [ \int_{a \in A} \pi_h(a \mid s) C(s, a) da ]$

$= E_{a ~ \pi_n} [(1 - I(s, a) C(s, a)] + E_{a ~ \pi_h} C(s, a)$

$\leq \kappa + \epsilon$.

This gives us our desired upper bound:

$S_{\pi_b} = E_{\tau∼P_{\pi_b}} \sum_{t = 0} \gamma^t C(s_t , a_t) \leq \sum_{t=0} \gamma^t (\kappa + \epsilon) = \frac{1}{1 - \gamma} (\kappa + \epsilon)$.

The same proof with $I(s, a)$ replaced by $M(s)$ would produce the same bound for Theorem 2 (this again depends on the slightly modified Assumption 2, where $\pi_b$ is replaced by $\pi_n$).

This bound is not particularly interesting – it basically says that if the two possible sources of failure are small for each individual step, then running it for many steps amplifies the (discounted) failures by a factor of at most $\frac{1}{1 - \gamma}$. The authors’ bound is slightly different because of the slightly different Assumption 2, but I do not see what additional insight it gives.

(Incidentally, the “testing safety guarantee” should be explained more clearly. I was unclear on what the theorem was even proving. Based on reading the proof, the rough claim is “if during training every state-action pair was explored, and the resulting learned policy always takes the action recommended by the human if any such action exists for the current state, then we have an upper bound on the discounted failures of that policy”.)


**Experimental issues**

_Comparisons to RL._ The baselines for comparison should primarily be other approaches using human data, rather than RL. It is not surprising that a human-in-the-loop approach can outperform RL: an RL algorithm typically has to solve the exploration problem, which can take significant time / steps, while many human-in-the-loop approaches (including PVP) can leverage the fact that the human guides the learning and has already solved exploration. (One striking example is that the hard-exploration game Montezuma’s Revenge can be solved using a single human demonstration [1], while being very challenging for most RL methods.)

_Discrepancy in the HACO baseline._  The RL results for MetaDrive-Keyboard are mostly copied from the HACO paper (incidentally, please note this in the caption of Table 1), suggesting that the setting is the same as in the HACO paper. However, the authors report that HACO only gets ~143 return with 40K training data, while the HACO paper claims that it gets ~349 return with 30K training data. Why is there such a big discrepancy?

_Lack of other baselines._ Apart from the one MetaDrive-Keyboard environment, there do not seem to be even simple behavioral cloning baselines for most environments. These are crucial: it is generally easier for humans to provide demonstrations rather than interventions, so it is important to check that the interventions are actually providing significant value on top of what could be achieved with demonstrations.


**Minor issues**

In Equation (4), s and a are unbound variables.

> We introduce two important assumptions on the human subject without imposing any constraint on the human policy.

This seems false – the next sentence introduces a constraint on the human policy (that when you sample actions from it, they are safe with high probability).

> We measure safety with the notation of cumulative discounted probability of failure as $S_{\pi}(s_t) = E_{\tau∼P_{\pi}} \sum_{t’ = t} \gamma^{t’-t} C(s_{t’} , a_{t’})$

> The expected discounted probability of failure $S_{\pi_b}$

$S_{\pi}$ is actually the expected discounted _number_ of failures, according to your definition, though this does also serve as an upper bound on the expected discounted _probability_ of failure.

[1] Salimans, Tim, and Richard Chen. "Learning montezuma's revenge from a single demonstration." arXiv preprint arXiv:1812.03381 (2018).
clarity,_quality,_novelty_and_reproducibility: Clarity: While the paper was somewhat wordy overall, it was reasonably clear.

Quality: As mentioned above, the experimental evaluation is lacking.

Novelty / originality: The method is relatively incremental, but this would be fine if I were convinced of the benefits.

Reproducibility: Didn’t evaluate in detail but it generally seems reproducible.
summary_of_the_review: The theoretical guarantees are not particularly significant, and the experimental evaluation is lacking (primarily due to (1) very few baselines that use human data, and (2) numerical discrepancies with the HACO paper suggesting poorly tuned baselines). As a result I recommend against acceptance.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1917/Reviewer_uDKA
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper presents "Proxy Value Propagation", a class of algorithms for learning from "active" human interventions, in the absence of an explicit reward function.  In this setting, during training a human observes all policy roll-outs in real time, and has the option to override the policy at any time and take control themselves.  In addition to the Bellman loss, PVP adds a secondary loss term that assigns large Q-values to actions the human took in states where they intervened, and small (negative) Q-values to the actions the learner's current policy would have taken in those states.  They argue that in addition to encouraging the learned policy to imitate the human's actions, it also encourages them to seek out parts of the state space where human demonstrations are available.

They evaluate PVP (combined with either TD3 or DQN depending on the environment), across several benchmark RL tasks, including autonomous driving in the CARLA simulator and the Atari Skiing environment.  In most of these environments (with the notable exception of the Atari environment) they showed a significant improvement in sample efficiency (and in some case final performance) relative to pure RL with task rewards, while also showing a significant improvement in final performance over other methods for learning from online and offline human demonstrations.
strength_and_weaknesses: The main strength of the paper is the strong performance of the PVP algorithm when compared with the set of baseline approaches.  Their experimental results are also fairly extensive given that most of the experiments are performance in real time with human subjects.  PVP showed a dramatic improvement over baseline RL algorithms in terms of sample complexity, which is not surprising given that human demonstrations provide better exploration and a much denser reward signal.  More surprising is it strong performance in terms of the final policy when compared to other human-in-the-loop methods, and against imitation learning.

In terms of weaknesses, the most significant is Theorem 2.  While it appears to be technically correct (proofs haven't been thoroughly checked), the actual guarantee is very weak.  Specifically, the coverage term xi could be arbitrarily small.  If the test policy (or "novice" policy) has a small probability of visiting states in which the human previously intervened, then the failure bound would become trivial.  Even if the human provides demonstrations for most of the relevant parts of the state space, there is nothing in Theorem 2 that would guarantee that the novice policy would actually reach these states.  More fundamentally, it isn't clear what the constraints on the novice policy are, and whether it is actually required to be an optimal strategy given the reward derived from human demonstrations.  Furthermore, the bound in Theorem 2 would seem to only be practically relevant in finite state spaces.

Another, somewhat less serious weakness, is that some of the claims about how humans behave in this human-in-the-loop setting are not well supported.  In particular, the claim that removing random exploration from roll-out policies improves user experience isn't really supported by their experiments.  They don't directly evaluate user experience, just sample efficiency and task performance, and the do not do ablations comparing PVP with and without random exploration. They also claim in section 3 that the lack of human intervention indicates that the learned policy is behaving correctly, while humans only intervene when behavior is incorrect.  While this makes sense at the level of individual actions in certain environments, we could imagine environments where a human is unlikely to intervene unless they have already observed that the learned policy is capable of even getting near some goal.  It could also be the case that humans are likely to intervene when the agent is close to the goal, in order to provide the most informative demonstrations.  The authors could devote more space to discussing the difficulty of interpreting human behavior, and the extent to which human teaching behavior might change in environments other than the ones (primarily autonomous driving) considered in their experiments.
clarity,_quality,_novelty_and_reproducibility: Here are some specific issues that should be addressed for a camera-ready version of the paper:
- There would seem to be a risk that assigning reward to actions (and therefore states) where humans have intervened could lead to spurious positive reward cycles, or could even lead to policies that intentionally get into difficult parts of the state space to elicit human intervention.  Such issues should at least be discussed in section 3 (or elsewhere as appropriate).
- Could be made a little more clear that "active" setting means that humans can intervene during roll-outs, rather than generating demonstrations beforehand, or evaluating roll-outs after the fact.  It should be made clear that there is nothing preventing this approach from being used in an "asynchronous" setting, where sampling and training are occurring at different times.
- Should address the possibility that, in tasks with high action frequency, humans may intervene after a single mistaken action, even if later actions were correct.
- remove the unsupported claim that reducing noise actually improves user experience ("wellbeing"), or provide supporting evidence.
- figure 2 is difficult to understand, perhaps the idea could be captured better in the text, or through a figure with less text?
- make it clear what assumptions Theorem 2 makes regarding the optimality of the novice policy being evaluated at test time, and explain the limitations of the theoretical results (particularly, that Theorem 2 becomes vacuous in environments with continuous state spaces).
- explain how the baseline RL algorithms were tuned.  In figure 4, the sub-figure for DQN over 10 million steps suggests that performance on the Atari task actually drops, suggesting a problem with the choice of parameters.  How does this result compare to the state-of-the-art for this Atari game?
summary_of_the_review: The decision to recommend acceptance is largely based on the fact that the work presents strong performance in the relatively underexplored setting of learning with active human involvement (as opposed to learning from batched demonstrations, or post-hoc trajectory comparisons).  It is interesting that a relatively simple approach can perform well in practice, and can outperform arguably more sophisticated methods.

That said, there is significant room for improvement in the paper, though these changes could be made in a camera-ready version.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1917/Reviewer_6qMp
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The work attempts to highlight the benefits of human in the loop RL (terms it as active involvement) where the human acts as a safety guide to the RL agent by interrupting the agent’s actions (for safety) and provides action advice on the interrupted states. Their key insight is to assign low credit to actions (that the agent was going to execute) on states at which the human intervened and further attempt to get the agent to visit those states where the human has previously provided some action advice & then follow it.
strength_and_weaknesses: Strength : 

1. Overall the paper was clear and to the point. The problem was well motivated in the introduction & the coverage of related works was helpful. 

2. I really like the fact that actual human users were used for their studies, instead of passing oracles as “humans”. Moreso, the authors also mention “cognitive cost”, where although better measures of cognitive cost exists in HiL (Human in the loop) setups, I appreciate the authors effort to document these important measures in an HiL work.

3. The evaluation seems to be thorough to justify the benefits of the methods  [even though I have my concerns - refer weaknesses] but empirically I am convinced that the method is superior to other HiL baselines like HG-Dagger etc. 

4. The authors provide safety guarantees regarding probability of failure of the behavior policy and probability of failure of novice policy. [I would have liked for the authors to draw more insights from these theorems and connect them with the rest of the work]

Weakness : 

1. One major weakness I find with the work is with its motivation. From the beginning especially abstract & Introduction the work seems to be trying to learn human intent or preferences, however suddenly their methodology hinges on human interruptions for safety violations. Is the human a safety guide, or there to give preferences or is the assumption that the human preference is safety? 

2. For Goal 1, authors say, “... At the same time, the agent actions a_n \sim \pi_n should always have lower values than others since they are rejected by human subject…”. I can agree to the authors insight that if the human has interrupted at a state then there might be something unsafe about the action which was going to be executed. However why should the value of the action be lower than all other actions? It is possible that all other actions were equally unsafe. I would want the authors to clarify why is this a valid argument to make?

3. For the work’s second intuition - “... we should select actions at these states that lead to the transition to those states where human have previously intervened…”. Again I fail to understand why this should be the case. The human guidance was provided on states where they interrupted which the context being that an interruption most likely means that the action is unsafe at that state. I would further make the claim that, that state is more “unsafe” than other states [marginalizing over actions]. Then, even if the agent has an action-advice available (from the human) on this state, why should the agent want to transit back to this unsafe state? Just because agent has action-advice or maybe the best action choice available for a certain state doesn’t necessarily mean it should transition to that state [which ofcourse would be extremely suboptimal to begin with & can easily be unsafe].

clarity,_quality,_novelty_and_reproducibility: I have asked clarification questions in my review. Additionally, 

If this is infact a safety-via-human-advice style of work (instead of preference learning), does the argument that reward specification is a hard problem still apply? That is, can you describe to what extent, in the domains you have considered for your experiments, it is hard to specify safety violations explicitly [to justify an HiL approach]. 

Quality, Novelty & Reproducibility : 
1. I find that the text, supplementary material should be enough to reproduce the results shown in the manuscript.

summary_of_the_review: I find that the problem is quite interesting for safety in the RL line of work. The paper is well written with enough empirical results to showcase the method's effectiveness over HiL baselines. My major concerns lie with the motivation and the intuitions authors used to come up with their technique. I would like to hear from the authors to realize if I am missing some aspects of the work and review my score. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1917/Reviewer_RMcE
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper describes a method for (only) learning from human intervention. The basic principle is to set the Q value for "demonstrated" state, action pairs to 1 and for not-selected actions in these states to -1. Following is an update of the Q function, which is then used in a common value-based RL algorithm. Besides this algorithm, the contributions are an experimental evaluation, a replay buffer method for HiL RL algorithms and theoretical safety guarantees.
strength_and_weaknesses: The paper is well written and mostly easy to follow. The evaluation is quite good, because real, human trials have been performed.  The number of baseline algorithms is also sufficient, however, the number of experimental domains could be increase. Although, considering the human evaluation, this is not expected. 

The most substantial weakness of the paper is its embedding into related work. The authors consider the approach a "reward-free policy learning" approach, which is technically correct, but misses the point. The algorithm belongs to the category "learning from demonstration" and "learning from advice". This is important, because it embeds the work in a line of long standing work, that uses the same principle of updating Q Values based on demonstrated/advised actions. (like TAMER, works of Najar et al. and HACO) The HACO algorithm is mentioned, but needs to be discussed in detail, because it seems, the most substantial difference is using a fixed value instead of an increment to the Q value. The impact of this difference is very interesting, but sadly only part of the appendix. Furthermore, the authors state, that another reason for the much lower performance of HACO is the SAC algorithm. However, neither HACO, nor PVP depend on a specific RL-algorithm, and therefore it is easily to realize (and evaluate) either HACO with a deterministic algorithm or PVP with SAC. Furthermore, several parts of the evaluation have been moved to the appendix. At least the core findings should be part of the main paper.

Another smaller, weakness is goal 2. First, the assumption that one should try to reach human-involved states is questionable. E.g., consider the near-miss example. It is likely better to completely avoid these states, than to "replay" the near-miss resolution a human proposed. Second, the reasoning is mostly based on how the algorithm works, not the other way around.

I also want to point out, that the idea of "Balanced Buffers" is quite interesting and should be made more prominent.

Edit: The authors resolve several issues in the rebuttal, and the score reflects that.
clarity,_quality,_novelty_and_reproducibility: The paper is clear to understand and of high quality. Novelty is limited or at least questionable, due to the insufficiently discussed relation to already existing approaches. A reimplementation of the algorithm should be straight-forward, as complexity is limited. Relevant hyperparameters are given. Exact reproducibility is not expected, due to the human involvement. Additionally, there is one issue that needs to be clarified: Figure 5 shows a minor amount of interventions across the board. However, in the first iterations, interventions are expected to be dominating, because the initial (random?) policy is likely to behave very badly. In case the initial policy is more informed, this needs to be clarified - and would move the algorithm strongly into the area of learning with advice.

Small issue:
GAIL is found as BC in Table.1 (it seems, please clarify)
summary_of_the_review: Differences to HACO and other related work (from the domain of learning from demonstrations and advice) needs to be substantiated. Currently, it seems the idea is of limited novelty, which moves the focus of the paper to the evaluation. The evaluation is good, but not sufficient when only considering a variant of already existing ideas. Especially, because several, interesting findings are only found in the appendix.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold