Paper: Enhancing Human Experience in Human-Agent Collaboration: A Human-Centered Modeling Approach Based on Positive Human Gain (ICLR-2024)

Reviewer comment (Writing):
My main concern with this work is that its technical presentation is currently unclear to the point that it is not possible to determine which algorithm is actually being evaluated, yielding two very different possible explanations for the empirical results.

Author response:
**Q1**: Regarding the technical presentation.

**A1**: We have carefully revised the notations in the manuscript and rewritten Sec. 2.2, Sec. 3.1 and Sec. 3.2 to make the technical presentation clearer. All modifications we marked in red in the rebuttal version. We briefly clarify your confusion below. 

Our work aims to train agents to enhance the human experience while maintaining the agent's original abilities as much as possible. Therefore, our approach is essentially optimizing both $R$ and $R^H$, i.e., $J(\theta) = E_{\pi_{\theta},\pi_H} [G_t + \alpha \cdot G_t^H|s_t=s]$, corresponding to the self-centered optimization objective and the human-centered optimization objective, respectively, with the two objectives weighted by $\alpha$. 

During the optimization process, we found that the human-centered advantage $A_H(s, a) = E_{\pi_\theta, \pi_H}[G^H_t|s_t=s,a_t=a] - V_H^{\pi_{\theta}, \pi_H}(s)$ in the policy gradient approximation Eq.3 may result in human-agent credit assignment issues. For instance, rewarding agents for assistance when humans can effortlessly achieve their goals is unnecessary, as this could potentially lead to agents learning incorrect behavior and even losing their autonomy (see experimental results in Sec.4.2). 

Our key insight to address this challenge is that the human contribution, termed **human primitive value**, i.e., $V_H^{\pi_0, \pi_H}(s)$, should be distinguished from human rewards. $V_H^{\pi_0, \pi_H}(s)$ represents the expected human return for a given state $s$ under the setting of collaboration between the human $\pi_H$ and the pre-trained agent $\pi_0$. The residual benefits, representing the agent's actual contribution in enhancing human to achieve goals, are referred to as **human gains**, i.e., $\Delta(s, a)=E_{\pi_\theta, \pi_H}[G^H_t|s_t=s,a_t=a] - V_H^{\pi_0, \pi_H}(s)$. The agent is solely encouraged to learn effective enhancement behaviors, i.e., $A^{+}(s) = ${$a|a \sim \pi_\theta, \Delta(s, a) > 0$}. We propose the RLHG algorithm to actualize our insight. The RLHG algorithm initially learns a human primitive value network $V_{\phi}$ in the Human Primitive Value Estimation stage. In the subsequent Human Enhancement Training stage, the RLHG algorithm encourages the agent to learn effective enhancement behaviors (actions contributing to positive human gains). For those invalid enhancement behaviors, i.e., $A^{-}(s)=A(s) \setminus A^{+}(s) = ${$a|a \sim \pi_\theta, \Delta(s, a) \le 0$}, the human-centered advantage in Eq.4 is set to 0 and only the self-centered advantage exists to guide the agent's optimization.

Reviewer comment (Writing):
The way RLHG is currently presented suggests that the algorithm attempts to select AI actions that maximize the difference between the advantage of the human's own actions under the AI's action, and their advantage under some baseline AI policy (this advantage captured by the term $\hat{A}_H (s, a)$). Working through the derivation of the RLHG update, however, it appears that RLHG may essentially be equivalent to optimizing the weighted combination of the human and task rewards $R + \alpha R^H$. This may simply be a misunderstanding of the notation used, but in either event the presentation needs to be improved for a final version of the paper.

Author response:
**Q1**: Regarding the technical presentation.

**A1**: We have carefully revised the notations in the manuscript and rewritten Sec. 2.2, Sec. 3.1 and Sec. 3.2 to make the technical presentation clearer. All modifications we marked in red in the rebuttal version. We briefly clarify your confusion below. 

Our work aims to train agents to enhance the human experience while maintaining the agent's original abilities as much as possible. Therefore, our approach is essentially optimizing both $R$ and $R^H$, i.e., $J(\theta) = E_{\pi_{\theta},\pi_H} [G_t + \alpha \cdot G_t^H|s_t=s]$, corresponding to the self-centered optimization objective and the human-centered optimization objective, respectively, with the two objectives weighted by $\alpha$. 

During the optimization process, we found that the human-centered advantage $A_H(s, a) = E_{\pi_\theta, \pi_H}[G^H_t|s_t=s,a_t=a] - V_H^{\pi_{\theta}, \pi_H}(s)$ in the policy gradient approximation Eq.3 may result in human-agent credit assignment issues. For instance, rewarding agents for assistance when humans can effortlessly achieve their goals is unnecessary, as this could potentially lead to agents learning incorrect behavior and even losing their autonomy (see experimental results in Sec.4.2). 

Our key insight to address this challenge is that the human contribution, termed **human primitive value**, i.e., $V_H^{\pi_0, \pi_H}(s)$, should be distinguished from human rewards. $V_H^{\pi_0, \pi_H}(s)$ represents the expected human return for a given state $s$ under the setting of collaboration between the human $\pi_H$ and the pre-trained agent $\pi_0$. The residual benefits, representing the agent's actual contribution in enhancing human to achieve goals, are referred to as **human gains**, i.e., $\Delta(s, a)=E_{\pi_\theta, \pi_H}[G^H_t|s_t=s,a_t=a] - V_H^{\pi_0, \pi_H}(s)$. The agent is solely encouraged to learn effective enhancement behaviors, i.e., $A^{+}(s) = ${$a|a \sim \pi_\theta, \Delta(s, a) > 0$}. We propose the RLHG algorithm to actualize our insight. The RLHG algorithm initially learns a human primitive value network $V_{\phi}$ in the Human Primitive Value Estimation stage. In the subsequent Human Enhancement Training stage, the RLHG algorithm encourages the agent to learn effective enhancement behaviors (actions contributing to positive human gains). For those invalid enhancement behaviors, i.e., $A^{-}(s)=A(s) \setminus A^{+}(s) = ${$a|a \sim \pi_\theta, \Delta(s, a) \le 0$}, the human-centered advantage in Eq.4 is set to 0 and only the self-centered advantage exists to guide the agent's optimization.

Reviewer comment (Writing):
Looking at the RLHG policy gradient in equation 4, it appears that we can express the human gain advantage as: 
$$
     \hat{A}_H (s, a) = E_{\pi^H} [ G^{H}_t \vert s_t = s, s_t = a ] - V^{\pi^{e}_{\theta}, \pi^{H}}_H
$$
This is where the notation makes things unclear. As I understand it, $V^{\pi^{e}_{\theta}, \pi^{H}}_H$ is to be interpreted as a function of the state $s$ alone, in which case it would have no impact on the expectation of the policy gradient (though it would potentially reduce the variance).

Author response:
**Q1**: Regarding the technical presentation.

**A1**: We have carefully revised the notations in the manuscript and rewritten Sec. 2.2, Sec. 3.1 and Sec. 3.2 to make the technical presentation clearer. All modifications we marked in red in the rebuttal version. We briefly clarify your confusion below. 

Our work aims to train agents to enhance the human experience while maintaining the agent's original abilities as much as possible. Therefore, our approach is essentially optimizing both $R$ and $R^H$, i.e., $J(\theta) = E_{\pi_{\theta},\pi_H} [G_t + \alpha \cdot G_t^H|s_t=s]$, corresponding to the self-centered optimization objective and the human-centered optimization objective, respectively, with the two objectives weighted by $\alpha$. 

During the optimization process, we found that the human-centered advantage $A_H(s, a) = E_{\pi_\theta, \pi_H}[G^H_t|s_t=s,a_t=a] - V_H^{\pi_{\theta}, \pi_H}(s)$ in the policy gradient approximation Eq.3 may result in human-agent credit assignment issues. For instance, rewarding agents for assistance when humans can effortlessly achieve their goals is unnecessary, as this could potentially lead to agents learning incorrect behavior and even losing their autonomy (see experimental results in Sec.4.2). 

Our key insight to address this challenge is that the human contribution, termed **human primitive value**, i.e., $V_H^{\pi_0, \pi_H}(s)$, should be distinguished from human rewards. $V_H^{\pi_0, \pi_H}(s)$ represents the expected human return for a given state $s$ under the setting of collaboration between the human $\pi_H$ and the pre-trained agent $\pi_0$. The residual benefits, representing the agent's actual contribution in enhancing human to achieve goals, are referred to as **human gains**, i.e., $\Delta(s, a)=E_{\pi_\theta, \pi_H}[G^H_t|s_t=s,a_t=a] - V_H^{\pi_0, \pi_H}(s)$. The agent is solely encouraged to learn effective enhancement behaviors, i.e., $A^{+}(s) = ${$a|a \sim \pi_\theta, \Delta(s, a) > 0$}. We propose the RLHG algorithm to actualize our insight. The RLHG algorithm initially learns a human primitive value network $V_{\phi}$ in the Human Primitive Value Estimation stage. In the subsequent Human Enhancement Training stage, the RLHG algorithm encourages the agent to learn effective enhancement behaviors (actions contributing to positive human gains). For those invalid enhancement behaviors, i.e., $A^{-}(s)=A(s) \setminus A^{+}(s) = ${$a|a \sim \pi_\theta, \Delta(s, a) \le 0$}, the human-centered advantage in Eq.4 is set to 0 and only the self-centered advantage exists to guide the agent's optimization.

Reviewer comment (Theory):
The other interpretation, which is more consistent with the intuition behind the algorithm (and with equation 5), is that $V^{\pi^{e}_{\theta}, \pi^{H}}_H$ is really a function of both $s$ and the AI's action $a$ (or equivalently, $V : S \mapsto \Re^A$).

Author response:
**Q1**: Regarding the technical presentation.

**A1**: We have carefully revised the notations in the manuscript and rewritten Sec. 2.2, Sec. 3.1 and Sec. 3.2 to make the technical presentation clearer. All modifications we marked in red in the rebuttal version. We briefly clarify your confusion below. 

Our work aims to train agents to enhance the human experience while maintaining the agent's original abilities as much as possible. Therefore, our approach is essentially optimizing both $R$ and $R^H$, i.e., $J(\theta) = E_{\pi_{\theta},\pi_H} [G_t + \alpha \cdot G_t^H|s_t=s]$, corresponding to the self-centered optimization objective and the human-centered optimization objective, respectively, with the two objectives weighted by $\alpha$. 

During the optimization process, we found that the human-centered advantage $A_H(s, a) = E_{\pi_\theta, \pi_H}[G^H_t|s_t=s,a_t=a] - V_H^{\pi_{\theta}, \pi_H}(s)$ in the policy gradient approximation Eq.3 may result in human-agent credit assignment issues. For instance, rewarding agents for assistance when humans can effortlessly achieve their goals is unnecessary, as this could potentially lead to agents learning incorrect behavior and even losing their autonomy (see experimental results in Sec.4.2). 

Our key insight to address this challenge is that the human contribution, termed **human primitive value**, i.e., $V_H^{\pi_0, \pi_H}(s)$, should be distinguished from human rewards. $V_H^{\pi_0, \pi_H}(s)$ represents the expected human return for a given state $s$ under the setting of collaboration between the human $\pi_H$ and the pre-trained agent $\pi_0$. The residual benefits, representing the agent's actual contribution in enhancing human to achieve goals, are referred to as **human gains**, i.e., $\Delta(s, a)=E_{\pi_\theta, \pi_H}[G^H_t|s_t=s,a_t=a] - V_H^{\pi_0, \pi_H}(s)$. The agent is solely encouraged to learn effective enhancement behaviors, i.e., $A^{+}(s) = ${$a|a \sim \pi_\theta, \Delta(s, a) > 0$}. We propose the RLHG algorithm to actualize our insight. The RLHG algorithm initially learns a human primitive value network $V_{\phi}$ in the Human Primitive Value Estimation stage. In the subsequent Human Enhancement Training stage, the RLHG algorithm encourages the agent to learn effective enhancement behaviors (actions contributing to positive human gains). For those invalid enhancement behaviors, i.e., $A^{-}(s)=A(s) \setminus A^{+}(s) = ${$a|a \sim \pi_\theta, \Delta(s, a) \le 0$}, the human-centered advantage in Eq.4 is set to 0 and only the self-centered advantage exists to guide the agent's optimization.

Reviewer comment (Writing):
Some improvements to the presentation that could help clear things up:  
1. Specify the domain and range of the value functions in question  
2. Always condition value functions on their inputs ($V(s)$ instead of $V$)  
3. Clearly distinguish in the definitions between the AI actions and those taken by the human.

Author response:
**Q2**: Regarding other presentations.

> "Specify the domain and range of the value functions in question".

> "Always condition value functions on their inputs $V(s)$ instead of $V$".

> "Clearly distinguishing in the definitions between the AI actions and those taken by the human".

**A2**: Thank you for your suggestions, we have carefully revised the manuscript according to your suggestions in the rebuttal revision.

Reviewer comment (Experiments):
There is a question about the relative importance of the "human-gain" objective vs. the distinct human reward function $R^H$ in the empirical results. My understanding is that the baseline HRE fine-tuning process corresponds to simply maximizing the human's reward function, starting from the pre-trained Wukong agent. The fact that this performs so poorly would suggest that the human-gain objective is really the critical factor. Unfortunately, the HRE agent is not sufficiently well-defined.

Author response:
**Q3**: Regarding the description of the HRE.

**A3**: HRE (Human Reward Enhancement) essentially integrates the human reward directly into the policy gradient, which is Eq. 3. Based on your suggestion, we marked the gradient update of HRE when introducing the HRE agent in the experiment part, and we also supplemented the detailed optimization process of WuKong and HRE in Appendix C.1.

**Q4**: "Were an experiments conducted using RLHG with the task reward and human reward being equivalent (solely optimizing human agency)?"

**A4**: We conducted experiments to examine the influence of the balance parameter $\alpha$ ($\alpha$ is set to 0, 1, 2, and 4). Please see Appendix C.2 for detailed experimental results. 

**Q5**: "Do the HRE experiments correspond to maximizing the $\alpha$-weighted mixture of human and task reward?"

**A5**: Yes, the optimization objective of HRE is the same as RLHG, i.e., $J(\theta) = E_{\pi_{\theta},\pi_H} [G_t + \alpha \cdot G_t^H|s_t=s]$. And the difference between them is the policy gradient update (HRE: Eq.3, RLHG: Eq.4).

Reviewer comment (Reproducibility):
Explicitly writing down the gradient update or objective function for the HRE agent would further improve the reader's understanding of RLHG.

Author response:
**Q3**: Regarding the description of the HRE.

**A3**: HRE (Human Reward Enhancement) essentially integrates the human reward directly into the policy gradient, which is Eq. 3. Based on your suggestion, we marked the gradient update of HRE when introducing the HRE agent in the experiment part, and we also supplemented the detailed optimization process of WuKong and HRE in Appendix C.1.

Reviewer comment (Writing):
I have some questions about the notations. I think the notation in Sec.2.2 and Sec.3.1, 3.2 are a bit confusing and hard to follow. See my questions in the "questions" box.

Author response:
**A1**: We have carefully revised the notations in the manuscript and rewritten Sec. 2.2, Sec. 3.1 and Sec. 3.2 to make the technical presentation clearer. All modifications we marked in red in the rebuttal version. Below we provide clarification on the notation-related questions you raised.

Reviewer comment (Theory):
I think Dec-POMDP usually assumes that all agents maximize the same reward function. However, in this paper, the human and robot are not just optimizing the same reward function. Instead, it seems that the human is just a fixed policy that is not optimizing anything. The robot is a policy that optimizes the human gains ∆πH in Sec.3.2. Maybe it is worthy to clarify this.

Author response:
**A2**: We have rephrased the 2nd paragraph of Sec 2.2 and supplemented the description of the human policy to make it clear. "We define a human policy $\pi_H$, to be a mapping from observations $o^H \in \mathcal{O}^H$ to actions $a^H \in \mathcal{A}^H$, which remains fixed during the agent's optimization process and cannot be accessible to the agent."

Reviewer comment (Writing):
Due to this conflict, it did confuse me in the beginning and took me some time to figure out that in this paper, the human policy is fixed (e.g., learned from data), while the robot policy is learned as the "best-response" to the human policy. In other words, the human might not adapt to or might not be the best respond to the robot policy. It might be useful to make it clear.

Author response:
**A2**: We have rephrased the 2nd paragraph of Sec 2.2 and supplemented the description of the human policy to make it clear. "We define a human policy $\pi_H$, to be a mapping from observations $o^H \in \mathcal{O}^H$ to actions $a^H \in \mathcal{A}^H$, which remains fixed during the agent's optimization process and cannot be accessible to the agent."

Reviewer comment (Writing):
I think in multi-agent setting, the value function depends on both the robot policies, denoted by π or πθ, in this paper, and also the human policy πH. However, some notations in Sec.2.2 and Sec.3.1, 3.2 are only explicitly associated with one policy, rather than both the robot and human policies, which makes me confused.

Author response:
**A1.2**: Thank you for pointing this out, we have revised all equations in the manuscript by replacing  $E_{\pi_H}$ with $E_{\pi_{\theta},\pi_H}$.

Reviewer comment (Writing):
In the 2nd line of the paragraph before Eq.1, it says V^πθ, where I am confused that what is the human policy inducing this value function.

Author response:
**A1.1**：In the 2nd paragraph of Sec 2.2, we initially stated, "In agent-only team setting",  which does not include humans. In this setting, the optimization objective of the agent is to maximize the cumulative task reward for a given state $s$.  To eliminate possible ambiguities, we have rephrased this paragraph in the rebuttal revision as follows: "Most previous research work focuses on learning self-centered agents. In agent-only team settings, the agent is optimized to complete the task, with the optimization objective being to maximize the value function for a given state $s$, i.e., $V^{\pi_{\theta}}(s) = E_{\pi_{\theta}}[G_t|s_t=s]$, where $G_t=\sum_{k=0}^{\infty}\gamma^{k}R_{t+k+1}$ represents the discounted cumulative task rewards."

Reviewer comment (Theory):
In Eq.1's last term - expectation under πH, the total return Gt should depend on the robot policy and the human policy, but the expectation is only taken under πH, not (πH,πθ).

Author response:
**A1.2**: Thank you for pointing this out, we have revised all equations in the manuscript by replacing  $E_{\pi_H}$ with $E_{\pi_{\theta},\pi_H}$.

Reviewer comment (Theory):
Similar to Eq.2's last term - E_πH[...].

Author response:
**A1.2**: Thank you for pointing this out, we have revised all equations in the manuscript by replacing  $E_{\pi_H}$ with $E_{\pi_{\theta},\pi_H}$.

Reviewer comment (Writing):
Similar to the line below Eq.3: E_πH[...].

Author response:
**A1.2**: Thank you for pointing this out, we have revised all equations in the manuscript by replacing  $E_{\pi_H}$ with $E_{\pi_{\theta},\pi_H}$.

Reviewer comment (Writing):
Similar to Sec.3.2's 2nd paragraph's ∆π(s, a)=E_πH[...].

Author response:
**A1.2**: Thank you for pointing this out, we have revised all equations in the manuscript by replacing  $E_{\pi_H}$ with $E_{\pi_{\theta},\pi_H}$.

Reviewer comment (Theory):
I am also not sure how to get the last inequality in page number 4.

Author response:
**A1.3**：We have rewritten Sec. 3.2 to make our insights more readable and understandable. After modification, we removed the complex notation representation as well as the inequality.