Paper: Provably Efficient Learning in Partially Observable Contextual Bandit (ICLR-2024)

Reviewer comment (Presentation):
The problem setup is spread out into multiple sections and it takes the reader a long time to understand the main topic to be explored. I believe a dedicated section for setup will streamline the reader's understanding.

Author response:
In response to this feedback, we have made improvements to help readers understand the problem setup more efficiently. We have included a notation table in Appendix A, which provides a concise overview of the main notations used in the paper. This addition aims to assist readers in understanding the contributions and key concepts.

Furthermore, we have a preliminaries section that outlines the basic settings related to transfer learning. This section provides foundational information to establish a common understanding before delving into the specifics of the transfer tasks.

To maintain coherence and clarity, we introduce the main contents of each transfer task separately, rather than presenting them all at once. This approach allows readers to grasp the relevant information progressively and maintain a logical flow throughout the paper.

We appreciate the feedback and have taken steps to enhance the organization and presentation of the problem setup in order to make it more accessible to readers.

Reviewer comment (Novelty):
Some relevant references are missing. For example, there is a line of work including [1],[2],[3] which study the sequential problem under partial observation, characterized by graphs. I think a comparison is needed.

Author response:
In the new version, we cite your mentioned papers in the related work and compare the difference in terms of settings.

Reviewer comment (Novelty):
Since this paper is dealing with a subtle topic at the intersection of bandit, transfer learning, and partial monitoring, a thorough comparison with the existing work should be expected, detailing the connection and difference, which seems missing in the current version.

Author response:
We acknowledge the importance of comparing our work with existing literature in order to provide a comprehensive understanding of the field. While it is not feasible to compare every related work within the constraints of a 9-page limit, we have made efforts to provide relevant comparisons in terms of our approaches and theoretical results.

In the paper, we compare our causal bounds with those presented in [1]. We demonstrate the tightness of our bounds by showcasing the limitations of the bounds in [1] and provide a numerical experiment in Section 4 to support our claims.

Additionally, we compare our proposition 3.1 with the result presented in [2] and explain the practical implications of our findings. We also generalize the result in [3] through theorem 3.1 and compare the order dependence on policy space in our theorem 3.3 with [4], highlighting the intuitive improvement achieved.


[1] Bounds on causal effects and application to high dimensional data
[2] Partial counterfactual identification from observational and experimental data.
[3] Transfer learning in multi-armed bandit: a causal approach. 
[4] Bounding causal effects on continuous outcome.

Reviewer comment (Novelty):
The transfer learning problem for PO Contextual bandits is not well motivated.

Author response:
Transfer learning in partially observable contextual bandit settings is motivated by practical examples such as autonomous driving. In this scenario, human drivers possess experiential knowledge that is not captured by sensors alone. Works such as [1,2] have explored similar settings and investigated the problem.

Your mentioned "The expert summarizes ... to find the optimal policy faster" also serves as the motivation for this paper.

Reviewer comment (Writing):
The example on Page 3 is supposed to motivate, "direct approaches can lead to a policy function that is far from the true optimal policy". But I was not able to see this. How do you get the numbers 1.81? Rather, isn't the example motivating the need for considering U in the policy, if available?

Author response:
Thanks for pointing out the typo. We mistakenly added the do notation. 
The intention of the example was to demonstrate that directly transferring the conditional expectation E[Y|A,W] can be suboptimal in certain cases. Of course, having more information about the models will generally be helpful. It is necessary to consider the knowledge of U if it is available, but the way to use U should be carefully designed.

Reviewer comment (Theory):
The claim on sample complexity of the joint distribution F(a, y , w, u) given F(u) as well as F(a, y ,w) separately, is quite different from learning the full joint distribution from scratch. Therefore the comparison with existing sample complexity analysis is slightly misleading. Specifically, "Our regret demonstrates an improvement in dependence on policy space Π from P(|Π|) to P(log |Π|) compared with ...." is not entirely fair.

Author response:
The knowledge about the distribution only affects the width of the causal bounds. The key difference between existing methods [1,2] and ours lies in transferred online learning algorithms. As we explain, they treat each basis policy as an independent arm in bandit problems, which is often invalid as similar policies can provide information to each other. For example, the optimal policy and the second optimal policy typically yield similar actions and only differ in a small amount of contexts. This is the intuition behind how we improve this order dependence.

To address these challenges, we need to modify the classical IGW technique, which requires non-trivial modifications of its proof. We introduce the subspace of explored policies and show how the causal bounds can shrink its size. You can find the full proof in Appendix

Reviewer comment (Experiments):
The baselines considered are clear, but the setting is quite unclear. Specifically, is the setting considered general enough, or is it tailored to favor the algorithm proposed in the paper?

Author response:
The numerical setup is explicitly described in the main text, but we provides its details in Appendix A.6. For the first experiment, the values of distributions are randomly generated to provide a general setting. In the second experiment, some values are specifically chosen to demonstrate a significant improvement. This is because in certain cases where there is limited information about the distribution, the improvement achieved by the proposed algorithm may be negligible.

It is important to note that our method does not worsen performance with transferred knowledge. The numerical section of the paper demonstrates that naively transferring knowledge can lead to worse performance for classical algorithms. However, our proposed method does not suffer from negative transfer problems.

Reviewer comment (Writing):
In page 3 the authors say, "expert agent can observe the contextual variable in U,W". But also say in Figure 1 caption that "U is the unobserved context". Then why can any expert agent view this?

Author response:
U is unobserved to the learner but may be observed by the expert in certain cases. 
Actually, how the learner obtain the required information is not the key factor of our paper.

Reviewer comment (Writing):
Page 4 para 2, what is "skewness from F(u)"?

Author response:
It means P(U=0) differs a lot in P(U=1).

Reviewer comment (Writing):
In equation 1, can you specify what sets the sup and inf are over?

Author response:
sup/inf is taken with respect to all compatible causal models,
equivalently, all the joint distributions which satisfy the observations.

Reviewer comment (Writing):
Consider changing: "...investigate partially observable contextual bandits, we found few papers focusing on transfer learning in partially observable contextual bandits." --> "...we found few papers focusing on transfer learning in this setting."

Author response:
We follow your suggestions and modify our paper accordingly.

Reviewer comment (Evaluation):
mainly select the arm 1--> mainly selects arm 1.

Author response:
We follow your suggestions and modify our paper accordingly.

Reviewer comment (Writing):
Can the notations in Equation 2 be simplified?

Author response:
We follow your suggestions and modify our paper accordingly.

Reviewer comment (Writing):
"sequentially solving linear programmings" --> sequentially solve linear programs".

Author response:
We follow your suggestions and modify our paper accordingly.

Reviewer comment (Presentation):
Given that ICLR will allow an additional one page, it is expected that the authors will incorporate necessary, important information currently trimmed or in Appendix.

Author response:
Thank you for your feedback. Following your suggestions, we utilize the additional one page allowed by ICLR to incorporate the necessary information, 
including a conclusion subsection and brief descriptions about appendix structure. 
The following content is added in the additional one page.

"
In this paper, we investigate transfer learning in partially observable contextual bandits by converting the problem to identifying or partially identifying causal effects between actions and rewards. 
We derive causal bounds with the existence of partially observable confounders using our proposed Monte-Carlo algorithms. 
We formally prove and empirically demonstrate that our causally enhanced algorithms outperform classical bandit algorithms and achieve orders of magnitude faster convergence rates.

There are several future research directions we wish to explore. 
Firstly, we aim to investigate whether the solutions to discretization optimization converge to those of the original problem.
While the approximation error can be exactly zero when all referred random variables are discrete, 
it is still unclear which conditions for general random variables and discretization methods can lead to convergence. 
We conjecture that this property may be related to the sensitivity of the non-linear optimization problem.

Lastly, we aim to extend our IGW-based algorithm to continuous action settings. 
IGW has been successfully applied to continuous action settings and has shown practical advantages in large action spaces. 
This extension may be related to complexity measures in machine learning.
"


"
In appendix A, we put the omitted information regarding the examples, algorithms, theorems, implementation details and numerical setup. 
We also put partial related work section due to the strict page limitation of ICLR. 
In appendix B, you can find all proofs about claimed theorems.
In appendix B, you can find some related materials about causal tools. 
In appendix D, we generalize our sampling method on general simplex more than the special one defined by the special optimization (1). 
In appendix E, we put a conclusion section and discuss the future work. 
"

Reviewer comment (Presentation):
The paper seems abruptly trimmed or cut during submission. There is no conclusion or discussion. For example, Table 1 should be after Theorem 3.4. Without providing context, it is too abrupt. Further, it seems that the table next to Table 1 does not have a proper caption. Increase arraystretch to represent the table better.

Author response:
We have made adjustments to improve the organization. 
Specifically, we have reordered the content for Equations 5 and 6. 
In the current version, a paragraph titled "Estimation Error" has been added at the end of this subsection. 
Additionally, we have placed Table 1 at the end of Section 3.
As for Fig. 2, we have adjusted the figures accordingly.

Reviewer comment (Presentation):
The organization near Eq 5 and 6 is awkward.

Author response:
We have made adjustments to improve the organization. 
Specifically, we have reordered the content for Equations 5 and 6. 
In the current version, a paragraph titled "Estimation Error" has been added at the end of this subsection. 
Additionally, we have placed Table 1 at the end of Section 3.
As for Fig. 2, we have adjusted the figures accordingly.

Reviewer comment (Presentation):
Causal Bounds should be a separate section with proper use of subsection/paragraph structure. Causal bounds can be described as irrelevant to the transfer learning problem. Current paragraphs “Causal Bounds” and “Sampling valid causal models” as a whole can be in the section and restructured.

Author response:
Firstly, making significant changes to the structure may not be allowed by the ICLR guidelines, 
as the final version of the PDF should not differ substantially from the original submitted version. 
Secondly, Subsection 3.1 already provides a comprehensive description of computing causal bounds,
as the content about MAB in this subsection is limited.
To enhance coherence and maintain a clear focus on the topic of transfer learning, 
we have to add some information about bandits at the beginning and end of Subsection 3.2. 
This will help establish the connection between causal bounds and transfer learning tasks, 
as the objectives of causal bounds are different vary across transfer learning tasks.
Since computing causal bounds is an intermediate step in our method, 
we include it as a subsection within the section "Transfer Learning via Causal Inference."

Our remaining concern raised from your review is whether the title "transfer learning in MAB" of Subsection 3.1 is suitable. 
If you have more suggestions, we are glad to accept.

Reviewer comment (Theory):
In Page 8 near the end, the argument about instrumental variables is unclear. If they rely on IV, you may argue that your method is free of IV requirement. It is a bit unnecessary to mention that finding IV is an open problem in academia (Economics?).

Author response:
We modify this sentence in the updated version. 
We actually want to show that our method does not rely on IVs as you mentioned in your reviews.

Reviewer comment (Presentation):
Adjust the two plots in Figure 2.

Author response:
We have made adjustments to improve the organization. 
Specifically, we have reordered the content for Equations 5 and 6. 
In the current version, a paragraph titled "Estimation Error" has been added at the end of this subsection. 
Additionally, we have placed Table 1 at the end of Section 3.
As for Fig. 2, we have adjusted the figures accordingly.

Reviewer comment (Theory):
The regret definition is written in terms of W only (not U). At each time slot, do you compete with a policy that has access to the true function $f^*$ and both realizations of W, U or you compete against policy that has access to $f^*$, W and take expectation over U?

Author response:
Q: The regret definition is written in terms of W only (not U). At each time slot, do you compete with a policy that has access to the true function and both realizations of W, U, or do you compete against a policy that has access to $f^*$, W, and take the expectation over U?

A: Since the learner cannot observe $U$, we compete against a policy that has access to $f^*$, W, and takes the expectation over U. The realizability assumption only needs to hold in the sense of taking expectation with respect to $U$.

Reviewer comment (Writing):
What is h(a) in table (1)?

Author response:
Q: What is h(a) in Table (1)?

A: h(a) is the causal upper bound solved by the optimization problem (1). Please refer to the MAB part in section 3.1 for more details.

Reviewer comment (Theory):
What is sup/inf in equation (1)? Is it either inf or sup, both will work? Do you mean that one will give an upper bound and the other a lower bound? This should be clearly stated. In the same paragraph you mention that (1) gives a bound on the causal effects. Causal effects between which variables? What is the mathematical formula for the causal effect to see that (1) gives an upper bound?

Author response:
Q: What is sup/inf in equation (1)? Is it either inf or sup, and both will work? Do you mean that one will give an upper bound and the other a lower bound? This should be clearly stated. In the same paragraph you mention that (1) gives a bound on the causal effects. Causal effects between which variables? What is the mathematical formula for the causal effect to see that (1) gives an upper bound?

A: Equation (1) means we can get upper and lower bounds of its object by solving corresponding maximization and minimization problems, respectively.
The object is the causal effect between action $A$ and reward $Y$ and both its upper and lower bounds are needed in the transfer learning algorithms.
Theorem A.1 in Appendix shows the mathematical expression of the object
and in the last sentence on page 4, we show the expression of the object in the discrete setting.

Reviewer comment (Experiments):
The works in [1,2,3] consider contextual linear bandits with known context distribution without observing the realization. Even though the setup is more limited, I believe the authors need to provide a comparison when the results of the paper are limited to the setups of [1,2,3].

Author response:
Q: The works in [1,2,3] consider contextual linear bandits with known context distribution without observing the realization. Even though the setup is more limited, I believe the authors need to provide a comparison when the results of the paper are limited to the setups of [1,2,3].

A: Thank you for your suggestion. We have added the mentioned papers to the related work section. Please refer to the updated version of our paper for more details.