Paper: Learning to Make Adherence-aware Advice (ICLR-2024)

Reviewer comment (Novelty):
I think the "**Theoretical reinforcement learning**" section should be expanded a bit, e.g., to be an independent background section. I think this part is the most relevant literature. It would be better if the paper could spend more texts explaining the connection and differences between the existing literature and the current work from a more technical perspective.

Author response:
Expanding the "Theoretical reinforcement learning" section:
    
- We thank the reviewer for the suggestion. We presented the connection with theoretical reinforcement learning in the section Main Result, where we explained two informal theorems and discussed the connection and difference between the existing literature in detail. We believe that delving into technical details in the literature review section would be a bit too early, as by that time we haven't yet introduced the model and setting, which are critical in understanding the technical differences. 
    
- Moreover, we'd like to point out that the contribution of this work is beyond the sample complexity bound from theoretical reinforcement learning. Rather, the primary intention for the paper is to develop a new human-AI model for the sequential decision-making context and to derive algorithmic and theoretical insights for this new problem. The main motivation for the theoretical part is that we want to showcase the structural insights of the model (we can learn the model effectively by leveraging the adherence level), and connections for related areas (reward-free exploration and provably solving constrained MDP).

Reviewer comment (Experiments):
It is stated in the experiment section that two testbeds are considered, including the Flabby Bird domain and the Car Driving domain. However, I cannot find the Car Driving results.

Author response:
Two testbeds of the Flabby Bird domain and the Car Driving domain:
- Sorry for the confusion. We mentioned in line 2 on page 8 that we defer the second experiment on Car Driving to the end of the appendix.

Reviewer comment (Theory):
The major concern about this work is that there is a gap between the result for $\mathcal{E}_1$ in Algorithm 1 and that for $\mathcal{E}_2$ in Algorithm 3. The dependence on $S$ is typically more important than $H$ in the reinforcement learning problem. As claimed by the authors, the sample complexity bound for Algorithm 3 is sharper in the dependence on S than for Algorithm 1. But $\mathcal{E}_2$ should be a harder problem than $\mathcal{E}_1$, and thus the sample complexity bound for $\mathcal{E}_1$ is expected to be better. Can the authors discuss why there is such a gap and whether it is possible to modify Algorithm 1 to have a better sample complexity bound matching the one for Algorithm 3 in terms of $S$?

Author response:
Regret bounds: In the informal version of Theorem 1 on page 4, we mention that $O(H^2S^2A/\epsilon^2)$ is a PAC (probably approximately correct) sample complexity bound, which is different from the sample complexity bound $O(H^3SA/\epsilon^2)$ of Algorithm 3 (in line 32 and page 4, for Algorithm 3 we use sample complexity instead of PAC sample complexity. This type of sample complexity without PAC is also known as the sample complexity for BPI (best policy identification)). For more details of how PAC sample complexity and sample complexity differs, we refer to the statement of Theorem 1 and Theorem 2. 

-- In fact, to have a more comparable result to the sample complexity of $O(H^3SA/\epsilon^2)$ in Algorithm 3, we can just apply UCB-VI (Azar et al., 2015) to the machine's MDP (introduced in page 2), by the well-known trick that turns regret bound into sample complexity for BPI, and one will get the same sample complexity bound of $O(H^3SA/\epsilon^2)$ as in Algorithm 3. Notice that by simply doing this, there is no extra performance gain because neither Algorithm 3 nor this UCB-VI leverages the structure of the model. 
 
-- A natural question is why we mention Algorithm 3 for $\mathcal{E}_2$ instead of UCB-VI for $\mathcal{E}_2$ given that they have the same sample complexity. This is because Algorithm 3 comes from Algorithm 2, which aims to solve the reward-free exploration (RFE) problem instead of BPI. Therefore, Algorithm 3 is a more well-rounded algorithm and it can also perform RFE given more episodes (see Theorem 2), while UCB-VI is not designed for RFE.

-- We believe that by leveraging the inherent property of our human-AI interaction model, one can tweak the parameters of Algorithm 1 and have a $O(H^2SA/\epsilon^2)$ sample complexity bound for BPI, and that would be more efficient than $O(H^3SA/\epsilon^2)$ of Algorithm 3 and UCB-VI. However, we choose to derive a better PAC sample complexity bound because the PAC sample complexity bound is a stronger result than the sample complexity bound for BPI.

Reviewer comment (Theory):
Eq.(1) is not clearly explained, especially the third case in Eq.(1). Can the authors provide some more explanation on how the probability of the third case is obtained?

Author response:
More explanations of Eq. (1). We gave some interpretation of Equation (1) on page 2 (above Definition 1). The first equation in (1) comes from our statement that if the machine chooses to defer, the human follows the default policy $\pi^\text{H}$. The second equation in (1) comes from our statement that If the machine chooses to advise, the human takes the machine’s advice with probability $\theta(s,a^{\text{M}})$. The third equation comes from the fact that if the human agent chooses not to adhere to the machine's advice (this happens with probability $1-\theta(s,a^{\text{M}})$, which is defined as the adherence level), the human will follow its fixed human policy $\pi_h^{\text{H}}(\cdot|s)$. However, because $a^{\text{M}}$ is already suggested by the machine and the human agent chooses not to adhere, the resulting policy will have support on $\mathcal{A}/\{a^{\text{M}}\}$ with distribution $\pi^{H}_h(\cdot|s)/(1-\pi_h^{\text{H}}(a^{\text{M}}|s))$ (we divide by $(1-\pi_h^{\text{H}}(a^{\text{M}}|s))$ to normalize the distribution).

Reviewer comment (Experiments):
It would also be ideal to include an environment with continuous action spaces.

Author response:
"Does this method handle higher-dimensional environments or continuous action spaces?"
    
- First, we note that we are presenting a model, not only a method. The model aims to provide a theoretical foundation so that it will be compatible with different extensions. To incorporate higher-dimensional environments, our model is compatible with the existing theoretical analyses of learning methods involving linear function approximation of the state space which provide a solution for high dimension environments. For continuous action space, we have to develop a continuous model separately. Just like the case in generic RL, the related theoretical property for continuous action is similar to the discrete one but it is out of the current scope of this work.

Reviewer comment (Experiments):
The experimental setting also buckets all states into 2 coarse groupings of adherence levels for the advice-taking policy. In practice, the adherence levels are likely more complex and also time-varying. It would improve the paper to include experiments with a co-adapting advice-taking policy.

Author response:
"How does this method handle more complex adherence functions?"
    
- We first note that the proposed model can be naturally related to other theoretical extensions to solve the aforementioned problem. If the adherence is nonstationary, the problem can be solved by learning the adherence level at every time step, which is a standard extension for RL algorithms. If the adherence level changes from episode to episode, we can rely on a standard approach by introducing a variational budget (Besbes, et. al. 2014) for the adherence level over episodes and get the corresponding theoretical result. If the variation of human adherence level exceeds the variational budget, the corresponding decision-making problem remains an open problem for the general theoretical RL/online learning community.

Reviewer comment (Experiments):
Does this method handle higher-dimensional environments or continuous action spaces?

Author response:
"Does this method handle higher-dimensional environments or continuous action spaces?"
    
- First, we note that we are presenting a model, not only a method. The model aims to provide a theoretical foundation so that it will be compatible with different extensions. To incorporate higher-dimensional environments, our model is compatible with the existing theoretical analyses of learning methods involving linear function approximation of the state space which provide a solution for high dimension environments. For continuous action space, we have to develop a continuous model separately. Just like the case in generic RL, the related theoretical property for continuous action is similar to the discrete one but it is out of the current scope of this work.

Reviewer comment (Theory):
How does this method handle more complex adherence functions?

Author response:
"How does this method handle more complex adherence functions?"
    
- We first note that the proposed model can be naturally related to other theoretical extensions to solve the aforementioned problem. If the adherence is nonstationary, the problem can be solved by learning the adherence level at every time step, which is a standard extension for RL algorithms. If the adherence level changes from episode to episode, we can rely on a standard approach by introducing a variational budget (Besbes, et. al. 2014) for the adherence level over episodes and get the corresponding theoretical result. If the variation of human adherence level exceeds the variational budget, the corresponding decision-making problem remains an open problem for the general theoretical RL/online learning community.

Reviewer comment (Experiments):
How does this method handle co-adapting advice takers?

Author response:
"How does this method handle co-adapting advice takers?"

- As we understand, co-adapting means that through interaction, the human will adapt to the machine and change its behavior policy or adherence level. This corresponds to a learning problem where the underlying MDP is changing, and if we are not controlling the amount of change in the behavior policy/adherence level, there is no model/algorithm that can provide theoretical guarantees for the co-adapting setting. We understand that some data-driven methods without theoretical guarantees can be very effective and practical for this scenario. We agree with the reviewer that this is an important future direction, and we will discuss this point more in the future version of our paper.