Paper: On Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems (ICLR-2024)

Reviewer comment (Novelty):
It seems that the proposed algorithms are designed based on standard techniques, such as pessimism and MLE. The authors should elaborate more on their technical novelty. This is my main concern.

Author response:
We appreciate you being clear about your main concern. In fact, the goal of this paper is **not** to propose new algorithms but to propose a theory that seeks to explain the superior performance of preference-based methods that people observe in practice. We theorize this is because human rating feedback is less robust to bias than preference feedback, so we propose a new model for it, which we believe is closer to what actually happens in practice than existing models. New theoretical results for these classic algorithms are derived under our new observation model. (As can be seen, this model considers a new type of bias and noise different from the existing results in the literature, and the standard LCB algorithm behaves quite differently under our new model. For example, it would normally converge under partial coverage defined in Assumption 1, but we have shown in Section 4.1 it would not converge under our model.)

Reviewer comment (Theory):
It would be more clear to present conditions 1, 2 and 3 as assumptions. The authors should justify more on these assumptions. For example, why is condition 1 reasonable? Why the noise never changes the human preference?

Author response:
You’re correct that Conditions 1-3 are assumptions on the human rating model. We can make it clearer in our revision. Your question about Condition 1 is an important one. It is more thoroughly discussed in the paragraph starting with “Apparently” in the middle of Page 4. In short, Condition 1 is necessary because it keeps the estimation problem statistically consistent or well-defined, i.e., there exists an estimator that converges to the optimal policy in probability. In fact, this is an important contribution of our human rating model in Equation (3) over the ones in the existing literature, which do not always admit a solvable problem. In addition, Condition 1 is in alignment with the assumption of the BTL model (human preference) that humans can rank two state-action pairs correctly in expectation, making the comparison between human rating and preference fairer, as explained in Remark 1 on Page 4. 

As to why the noise never changes the human preference, we suppose you’re asking why the bias is not in the BTL model for human preference. This is indeed an important question in our theory. It might seem artificial at first to consider bias in the model for human ratings while keeping the BTL model for human preferences unaffected by bias. This is why we include Section 5, in which we consider bias in the BTL model too and find that using human preferences does not have a better sample efficiency than using human ratings in this setting, contrary to what happens in practice. Section 5 shows the superior performance of the human preference approach does not happen because its algorithm is more statistically efficient than the algorithm used with human ratings (standard LCB), but it is likely that this performance difference is caused by a difference in model assumptions between the two approaches (i.e., one type of model/data is easier than the other). This justifies why we theorize the bias affects human ratings but not so much human preferences. We will be happy to strengthen this discussion in our revision.

Reviewer comment (Theory):
In Theorem 1, the setup $C^*=2$ seems too specific. Can the result be extended to the one that allows general $C^*$ and depends on $C^*$?

Author response:
This is a good point. $C^\star = 2$ is actually not crucial in our lower bound construction; it can be any constant strictly greater than 1 and would only affect the final suboptimality by a constant factor. In fact, $C^\star = 2$ implies the data have very good coverage (the distribution shift is no more than a factor of 2 over all state-action pairs). The algorithm would only incur a worse constant lower bound under distribution shift with larger $C^\star$, which proves our point even better. Since the goal of this theorem is just to establish the suboptimality is a constant and does not decay as sample size increases, rather than to delineate a specific suboptimality rate, the current presentation is sufficient to demonstrate our point.

Reviewer comment (Theory):
Given the assumptions on the bias transform \( h \), I am not surprised that the preference-based feedback is rather invariant to such biases. So I am not sure if the final results are uncovering an informative phenomena.

Author response:
Your question is completely valid. The theoretical results are not meant to be surprising. There are two reasons for this. First, it is already a common belief that it is more accurate for human annotators to give preference feedback than to give rating feedback [1,2], but there hasn’t been any work that formalizes this theoretically. Furthermore, it is also notable that the bias in our rating model is monotone and does not change the optimal policy of the problem, so it is actually not immediately obvious why such bias makes learning from human ratings harder. Second, while it is not surprising that the robustness of human preferences to bias is a factor of why preference-based methods perform better in practice, we make an important contribution by pinpointing this factor as the likely primary cause among other factors. Specifically, we rule out the algorithmic factor by showing the pessimistic MLE used in preference-based approach is no more statistically efficient than the standard LCB used in rating-based approach.

[1] Touvron, Hugo, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 (2023).

[2] Kevin R. Tarlow, Daniel F. Brossart, Alexandra M. McCammon, Alexander J. Giovanetti, M. Camille Belle, and Joshua Philip. Reliable visual analysis of single-case data: A comparison of rating, ranking, and pairwise methods. Cogent Psychology, 8(1):1911076, 2021. doi: 10.1080/23311908.2021.1911076.

Reviewer comment (Novelty):
A number of prior works on contextual bandits with preference feedback is not mentioned. While the overall approach is sufficiently different (they optimize a least squared loss), I think they should be mentioned for completeness.  
  - Mehta, Viraj, et al. "Kernelized Offline Contextual Dueling Bandits." arXiv preprint arXiv:2307.11288 (2023).  
  - Dudík, Miroslav, et al. "Contextual dueling bandits." Conference on Learning Theory. PMLR, 2015.  
  - Saha, Aadirupa, and Akshay Krishnamurthy. "Efficient and optimal algorithms for contextual dueling bandits under realizability." International Conference on Algorithmic Learning Theory. PMLR, 2022.  
  - Bengs, Viktor, Aadirupa Saha, and Eyke Hüllermeier. "Stochastic Contextual Dueling Bandits under Linear Stochastic Transitivity Models." International Conference on Machine Learning. PMLR, 2022.  
  - Perhaps also: Bengs, Viktor, et al. "Preference-based online learning with dueling bandits: A survey." The Journal of Machine Learning Research 22.1 (2021): 278-385.

Author response:
Thank you for providing these references! We will include them in our revision.

Reviewer comment (Theory):
How would you go beyond tabular setting? Would you say the pessimistic MLE algorithm can be easily extended to say, a kernelized or linear rewards over a compact domain?

Author response:
There is no problem extending pessimistic MLE to a more general function approximation, as it has been studied in the linear function approximation setting by [1] and the general function approximation setting by [2]. However, how to extend our newly proposed rating model to the linear setting still needs investigation. Recall our model considers a general bias that is likely nonlinear in the real world. While a tabular algorithm is able to learn such biased reward, linear function approximation would suffer a huge approximation error; RKHS and more general function approximation like neural networks should be flexible enough to approximate such biased reward.

[1] Banghua Zhu, Jiantao Jiao, and Michael I Jordan. Principled reinforcement learning with hu- man feedback from pairwise or k-wise comparisons. arXiv preprint arXiv:2301.11270, 2023.

[2] Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D Lee, and Wen Sun. Provable offline reinforcement learning with human feedback. arXiv preprint arXiv:2305.14816, 2023.