Paper: Posterior Sampling via Langevin Monte Carlo for Offline Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
What is the difference between the convergence of Langevin Monte Carlo and the convergence of the RL posterior sampling under the offline setting? Will the former lead to the latter?

Author response:
To clarify the context, we assume that you are questioning the relationship between the Langevin Monte Carlo (LMC-PPS) way in Algorithm 2 and the pessimistic posterior sampling way in Algorithm 1. Note standard posterior sampling in [US21] does not have any frequentist guarantee. Please let us know if you are implying other stuff.  We are happy to elaborate on that more.

Responding to the relation between the two algorithms, we discussed this at length in the second paragraph of Section 6. For the reviewer’s convenience, we summarize it here. In our settings, LMC-PPS and exact posterior sampling (PPS with exact posterior samples, not with approximate samples via LMC) achieve the same guarantees. This shows a promising benefit of LMC-PPS, where we can employ first-order sampling methods (e.g., LMC) without compromising the statistical guarantees. This is immensely meaningful as, in many cases, exact posterior sampling is much more expensive to obtain (in our settings, as formally discussed in Section 6) or even intractable, while LMC is simply a noisy gradient-based method that can efficiently apply to any differentiable model.

Reviewer comment (Reproducibility):
Why is LMC, instead of SGLD, used in Algorithms 2 and 3? Can mini-batch data be used in simulations of the proposed algorithm?

Author response:
We can replace LMC with SGLD, where the use of stochastic gradients introduces additional complexities that are typically managed by the standard tools in LMC [1]. In essence, we only need to account for the additional source of stochasticity stemming from the random batch selection in the analysis. However, it's important to note that this point is ancillary to our main argument. Consequently, we introduce LMC for the sake of clarity in exposition.

In our simulations, we did, in fact, utilize mini-batch data (SGLD). Moreover, we discuss in Section D.2 in our appendix for a thorough description of the experimental setup and training particulars. We will enhance the elucidation of the relationship between LMC and SGLD in the final version.

Reference: 

[1] AS Dalalyan, A Karagulyan. User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient. Stochastic Processes and their Applications, 2019

Reviewer comment (Novelty):
The work in Uehara and Sun [US21] considers the setting where the representation of state-action $\phi(s,a)$ is unknown, whereas the submission assumes the feature representation function is known. I think it is not fair to claim the improvement from [US21].

Author response:
We apologize for the confusion.  We compared only with Section 8 of [US21], which studies the model-based Posterior Sampling for offline RL where there is no representation learning (the rest of [US21] is not related to posterior sampling). We will revise our paper accordingly.

Reviewer comment (Theory):
How is the regression objective related to posterior sampling? How is this a "Langevin" Monte Carlo method? It would be good to be introductory to Langevin Monte Carlo methods, and how the concepts are attached to the actual algorithm presented.

Author response:
The regression objective is the negative log-likelihood function of the posterior (see footnote 7 in our paper). If one simply samples from $\theta\sim \exp(-L_h(\theta))$, then it corresponds to the standard posterior sampling (with uninformative prior). However, direct sampling from the general $\exp(-L_h(\theta))$ could be intractable, so the alternative method would be adding noise to the gradient decent step of the negative log-likelihood function, which is essentially Langevin Monte Carlo (also check eqn (4) of [WT11]).

Reviewer comment (Novelty):
I had to understand the linear (or low-rank) MDP part very clearly before paying attention to the NTK function approximation part. I do not see any particular contribution from the NTK part to reinforcement learning theory. It is a good add-on result though.

Author response:
We study the statistical benefits of Posterior Sampling via Langevin Monte Carlo for offline RL in neural MDPs using NTK. Linear MDPs and neural MDPs (with NTK) are two different models that cover two different aspects of scenarios we might encounter. The guarantees in linear MDPs do not imply any guarantees in neural MDPs (in fact, roughly speaking, the bound for neural MDPs is worse than that for linear MDP by a factor of $\sqrt{H}$); thus, considering both settings gives two orthogonal views of the landscape of  Posterior Sampling via Langevin Monte Carlo for offline RL.

Reviewer comment (Experiments):
I wonder how this method performs on some offline deep-RL benchmarks.

Author response:
While the current scope of our paper focuses only on the theoretical understanding of the offline LMC algorithm, the empirical performances of our algorithm in large-scale scenarios remain an interesting and non-trivial question that we leave as future work.