Paper:
Scalar Posterior Sampling with Applications
We propose a practical non-episodic PSRL algorithm that unlike recent state-of-theart PSRL algorithms uses a deterministic, model-independent episode switching schedule. Our algorithm termed deterministic schedule PSRL (DS-PSRL) is efficient in terms of time, sample, and space complexity. We prove a Bayesian regret bound under mild assumptions. Our result is more generally applicable to multiple parameters and continuous state action problems. We compare our algorithm with state-of-the-art PSRL algorithms on standard discrete and continuous problems from the literature. Finally, we show how the assumptions of our algorithm satisfy a sensible parametrization for a large class of problems in sequential recommendations.

Review:
This paper scales up the previous posterior sampling RL methods, such as Thompson sampling, by extending to continuous state and action space. A Bayesian-style regret bound is given, and the author shows theoretically that the proposed algorithm has the state-of-the-art efficiency in time and space, and enjoys the applicability to continuous state and action problems. Given the merits, the paper seems to be written in a hasty manner, which makes the reading experience tougher than I expect.   There are some technical concerns 1. Assumption 2 seems to be restrictive to Gaussian noise. If the assumption is violated for non-Gaussian noise, what will happen and how robust is the algorithm?  2. One major concern is that the author claims that the proposed algorithm can scale up to continuous state/action space problems. However, in the experiments, there are only discrete action domains, which makes the comparison study less convincing. 3. The authors state that result is more generally applicable to continuous state-action problems; when dynamics of the system is parameterized by a scalar, their regret scales with the square root of the number of parameters and is independent of the number of states. This is opposite to the general sense. Since regret usually increases with the number of states. This paper assumes the MDP model is parameterized by a scalar, the parameterization of MDP is related with the number states. the regret is proportional with the complexity of the model parameterization. 4. Other methods for comparison study. The paper compares through simulations the performance of DS-PSRL algorithm with the latest PSRL algorithm called Thompson Sampling with dynamic episodes (TSDE). Some other algorithms should be compared with the proposed algorithm.   Some minor suggestions: pp2. The introduction is too long. A ‘Related work’ section should be split to formulate a separate section. line 169, ‘differential’ —> ‘differentiable’ line 192-194 the reference format of many places is incorrect. such as this place: there should be brackets of [Ouyang et al.] line 240-241 the TSDE work by Ouyang et al. should be introduced as early as the ‘Related work’ part. line 286: it should be “The algorithm provides generalization.”