Paper: Non-stationary Contextual Bandit Learning via Neural Predictive Ensemble Sampling (ICLR-2024)

Reviewer comment (Experiments):
The paper lacks a clear comparative analysis between PS and NeurPS. It would be beneficial for the readers if the authors could elucidate the apparent differences between the two.

Author response:
In this response, we address the question from the first three questions from the reviewer. The main advantage of NeuralPES is to offer a practical neural network based non-stationary context bandit algorithm that follows the key insight from PS, which is to only explore for enduring information in the environment. The key differences are:

     a) NeuralPES only requires one-step rollout into the future instead of PS which requires sampling an infinite sequence of future rewards for all actions.
     b) NeuralPES conditions on the one-step rollout of future world parameter prediction with a sequence model instead of sampled rewards, such that we can scale to a large space of contexts and actions.
     c) PS does not support neural networks and assumes a known world dynamics for nonstationarity, but NeuralPES doesn’t require a known world dynamics and can adapt to any type of nonstationarity.

Reviewer comment (Evaluation):
The authors mention that PS "suffers from their scalability" and "it does not efficiently scale", however, the specifics of these issues are not clearly outlined. I would recommend providing concrete examples or explanations to support these statements.

Author response:
In this response, we address the question from the first three questions from the reviewer. The main advantage of NeuralPES is to offer a practical neural network based non-stationary context bandit algorithm that follows the key insight from PS, which is to only explore for enduring information in the environment. The key differences are:

     a) NeuralPES only requires one-step rollout into the future instead of PS which requires sampling an infinite sequence of future rewards for all actions.
     b) NeuralPES conditions on the one-step rollout of future world parameter prediction with a sequence model instead of sampled rewards, such that we can scale to a large space of contexts and actions.
     c) PS does not support neural networks and assumes a known world dynamics for nonstationarity, but NeuralPES doesn’t require a known world dynamics and can adapt to any type of nonstationarity.

Reviewer comment (Novelty):
It seems that PS (Liu et al. 2023) operates on an infinite reward sequence for decision-making, yet this element is not present in the current paper. It would be interesting to know how this differs or if it is similar. Either way, a discussion on this point seems necessary.

Author response:
In this response, we address the question from the first three questions from the reviewer. The main advantage of NeuralPES is to offer a practical neural network based non-stationary context bandit algorithm that follows the key insight from PS, which is to only explore for enduring information in the environment. The key differences are:

     a) NeuralPES only requires one-step rollout into the future instead of PS which requires sampling an infinite sequence of future rewards for all actions.
     b) NeuralPES conditions on the one-step rollout of future world parameter prediction with a sequence model instead of sampled rewards, such that we can scale to a large space of contexts and actions.
     c) PS does not support neural networks and assumes a known world dynamics for nonstationarity, but NeuralPES doesn’t require a known world dynamics and can adapt to any type of nonstationarity.

Reviewer comment (Novelty):
The theoretical analysis may appear almost identical to that of (Liu et al. 2023). Are there any differences that the authors could highlight?

Author response:
The theoretical analysis differs in that (a) we now consider contextual bandits, and (b) generalizes our analysis to linear bandits.

Reviewer comment (Experiments):
How much execution time is required for NeuPS's learning and inference (or decision-making)? Considering the time-intensive nature of neural network training and inference, and even more so for ensemble models (depending on the value of M), it would be important to discuss the scalability implications of the execution time.

Author response:
Execution/inference time for NeuralPES is 2x of standard neural networks since it requires a prediction of one-step rollout of world parameters. Training-wise, it is approximately 3x to standard ensemble neural networks that carries out Thompson-sampling-like exploration per batch of data given that the model has three stages. However, in production environments, people usually need to completely retrain models to adapt to nonstationarities and hence if we can incrementally train from the current model using NeuralPES, it would actually save a lot of resources.

Reviewer comment (Experiments):
Given that the authors are using an "A100 40GB GPU", if the learning time is long, this could significantly increase the cost, which is a critical factor for practical applications.

Author response:
A100 40GB GPU was the only GPU available to us. From our experiment, any GPU with 2 GB is sufficient for the experiment.

Reviewer comment (Theory):
The authors state that "the regret of LinPS is zero" and "LinPS achieves 0 regret". Is this regret referring to long-run average regret?

Author response:
We refer to both the cumulative regret and the long-run average regret. We will revise those sentences to make it clear.

Reviewer comment (Theory):
The paper mentions that the algorithm "is such optimal". It would be helpful if the authors could clarify what they mean by "optimal", as many algorithms can achieve a long-run average regret of 0.

Author response:
In this particular scenario, the cumulative regret is zero, so the algorithm is optimal.

Reviewer comment (Experiments):
How is the ensemble size M set? Is it adjusted for each experiment? Knowing the effect of its size on reward changes and computation time seems necessary for demonstrating scalability.

Author response:
We select M = 10 according to [1] given that a 10-particle ensemble with prior functions is sufficiently enough to approximate the posterior distribution of the reward.  Increasing to 100 or even 1000 particles does not significantly improve the performance.

Reviewer comment (Experiments):
For the long-term experiments, how is the data processed? The authors mention grouping every 12 hours of recommendation into a contextual bandit format. Does this mean that the order of recommendations presented within the 12-hour period is ignored? If so, could this introduce bias?

Author response:
Yes the order of recommendations presented within the 12-hour is ignored but as we do not retrain our model within the 12 hours, it would not introduce any bias since we use the same NeuralPES model to address the whole batch of 12-hour data.

Reviewer comment (Reproducibility):
Despite the major contributions of the paper appearing to lie in the experimental section, the experiments are not written in a reproducible manner. The code is not shared, and it seems that there are parameters, such as the value of M, that are not mentioned in the paper.

Author response:
We are limited by our organization’s policy to not share code. We are in the process of obtaining permission to share open-source code. However, all hyperparameters and datasets are publicly available as listed in the paper.

Reviewer comment (Experiments):
One major weakness is that I find it hard to evaluate the significance of the theoretical results presented in this paper. It appears that the authors have only compared the performance of the LinPS algorithm against TS in a non-stationary environment, which is restricted to linear contextual bandits setting and also does not really reflect the optimality/near-optimality of the algorithm.

Author response:
You raised the point that our analysis is “restricted to linear contextual bandits,” which is indeed a fairly large class of problems, and can be a good starting point.

Reviewer comment (Theory):
It is unclear how the non-stationarity of the environment is defined. Does the proposed algorithm deal with all kinds of non-stationarity? Related to the point I raised above, the theoretical results that the authors provide are only for environments with abrupt change or AR(1) type of changes. But the paper positions itself in a way that suggests the algorithm can deal with any kind of non-stationarity.

Author response:
Our algorithm does not rely on any assumptions of how the rewards are generated. Our theoretical results are introduced with the purpose to provide more intuition on how our algorithm works. Our result (Corollary 1) is general, and we establish a result in AR(1) bandit (Corollary 2) only to provide more intuition on this particular example. We will add a discussion/note on theoretical results.

Reviewer comment (Presentation):
The presentation of the algorithms also make it difficult for readers to comprehend what each component is designed for. Currently there are 5 algorithms in the paper and the relationship among these algorithms is unclear. For example, what is TrainNN used as part of Algorithm 4? What are the connections between all of the algorithms? Why are you presenting Neural Ensemble Sampling before NeuralPES and can you highlight the differences?

Author response:
Thank you for your suggestions on improving the presentation of the paper. Algorithm 2 is a submodule of Algorithm 1 and Algorithm 1 makes a call to Algorithm 2. Similarly, Algorithms 3 and 5 are submodules of Algorithm 2.

Reviewer comment (Theory):
Could you provide more discussions that help readers understand the regret bounds in Theorem 1 and Corollary 1? For example, I am not sure what is $\mathbb{I}(\theta_2; \theta_1)$ in the regret bound of THM 1. In corollary 1, please also elaborate on what the entropy term represents.

Author response:
Thank you for bringing this up. We use $\mathbb{I}$ to denote mutual information, and $\mathbb{I}(\theta_2; \theta_1)$ corresponds to the mutual information between $\theta_2$ and $\theta_1$. In Corollary 1, we use $\mathbb{H}(p)$ to denote the entropy of a $\mathrm{Bernoulli}(p)$ random variable. We will include definition of mutual information and entropy in our manuscript.

Reviewer comment (Theory):
I am also unsure of the statement that says when $\theta_t$ changes very quickly, “then the regret of LinPS is zero that LinPS achieves optimal”. Could you elaborate? From the regret term defined in Definition 3, it seems that the benchmark that the algorithm is compared against is the best arm at every $t$. How could the regret be zero when the changes are even more frequent?

Author response:
The regret is always non-negative, so when the regret is zero, it implies that the algorithm is optimal. We would like to clarify that the benchmark that the algorithm is compared against is not “the best arm”, i.e., $\arg\max_{a \in \mathcal{A}}\mathbb{E}[R_{t+1, a} | \theta_{t+1}]$, but $\arg\max_{a \in \mathcal{A}}\mathbb{E}[R_{t+1, a} | \theta_t]$ instead. 

Therefore, when $\{\theta_t\}$ for ${t \in \mathbb{N}}$ is i.i.d. (an example when changes are frequent),  $\mathbb{E}[R_{t+1, a} | \theta_t] = \mathbb{E}[R_{t+1, a}]$ and it makes sense that regret can be zero.

Reviewer comment (Theory):
I am not very convinced by the exploration effectiveness of ensemble networks. The exploration comes from randomly drawing neural models, but it doesn't consider the estimation confidence interval of a single neural model like UCB or TS. It looks like an ensemble of greedy models.

Author response:
An Ensemble of neural networks can perform Thompson sampling-like exploration. This is also discussed in literature Lu and Van Roy (2017) [1], Qin et al. (2022) [2] and Osband et al. (2016) [3].

Reviewer comment (Experiments):
The training cost is too huge for this approach. In linear bandits, the training of linear models can be trained quickly. But for neural models, it cannot work. For one neural model, the training cost is already huge, but the algorithm needs to train a set of neural models in each round. Especially, it is for online learning scenarios. I don't think this algorithm can be scaled to large systems in practice.

Author response:
Neural contextual bandit algorithms are very commonly adopted in research that studies scalability [4, 5] and real-world applications [6], and also shown to be very incremental in terms of both training and inference [7]. Since each particle neural network is independent from others in an ensemble, in production systems, ensemble training usually leverages distributed GPU training and hence in general on par with standard deep supervised learning approaches. Since each ensemble particle is identical to a deep supervised learning model, inference cost is identical.