Paper: Provable Representation with Efficient Planning for Partially Observable Reinforcement Learning (ICLR-2024)

Reviewer comment (Writing):
The title and abstract is quite vague and overly broad – basically it just said it is about a theoretical framework on RL or planning (I am confused which one) in POMDPs.

Author response:
* RL vs. Planning 

  In RL,  the model learning refers to estimating the transition and reward,  and planning is referring to finding the optimal policy with the given or learned transition and reward (as explained in the Introduction section). We rigorously follow these definitions, where in our paper the algorithm **learns** a proper representation during the learning phase that can be used to perform planning (finding the optimal policy) easily.

Reviewer comment (Experiments):
The claim “state-of-the-art” is also too strong, as obviously Dreamerv2 and DrQ-v2, published in 2021, are no longer SOTA.

Author response:
* Terminology confusion 

   * "State-of-the-art”

      We have softened our claim and removed “state-of-the-art”, which is used for our competitors, not our method. 

We sincerely believe the paper should be reviewed by evaluating the contribution and significance, instead of criticizing the usage of terminology.

Reviewer comment (Experiments):
The claim also touches "offline POMDPs" but I did not see any results.

Author response:
* Offline POMDPs

  The pessimism mechanism in offline setting is similar to the optimism in the online setting, and the major difference between the optimism in online setting and pessimism in offline setting is that, in the offline setting we **minus** the same bonus from the reward while in the online setting we **add** the bonus into the reward. We already added the corresponding discussion in Appendix D in our updated version.

Reviewer comment (Novelty):
This work does not have much technical significance since it is heavily relied on recent work (Ren et al, 2023a).

Author response:
* Novelty and Significance comparing to Ren et al. (2023a)

  We strongly disagree with this claim about the novelty and significance of our work.  Based on our personal communication with the authors of Ren et al (2023a), how to extend LV-Rep to POMDP is “extremely difficult, which cost them one year already”, as we also discussed in Section 4.  
  The extension is highly nontrivial and significant as we discussed in Section 3, especially considering the research on planning in POMDPs has stuck for decades. We emphasize the simplicity of the algorithm should be considered as an advantage, which enables the LV-Rep to more practical settings to handle partial observations, rather than a drawback.

Reviewer comment (Writing):
The paper talks about “low-rank POMDPs”, but no definition is provided. How is it connected to L-step decodability?

Author response:
* Low-rank POMDPs

  The low-rank structure is indicating the latent variable factorization can be finite, or even if it is infinite, the eigendecay speed is fast, as we discussed in Appendix. The L-step decodability property ensures the rank is smaller than the exponential of the horizon, as discussed in [3].

Reviewer comment (Experiments):
How partially observable is in the visual control benchmarks? As they were also tackled by Markovian methods.

Author response:
* Non-Markovian in Visual Control 
  
  In the image-based visual control benchmarks, although we can have the images, the control states, e.g., velocity, can not be inferred from one single image. There is no doubt that the Markovian methods can be applied for these tasks, but the performances will be significantly degenerated, due to the missing of important information. 
  To further demonstrate the non-markovian property, we tested the algorithms with markovian assumption (L=1) vs non-markovian (L>1) (Figure 6 of Appendix H.1). As we can see, the non-Markvoian algorithm performs significantly better.

Reviewer comment (Writing):
The presentation could be improved. Some technical details are inconsistent or lack an appropriate definition (see detailed comments below). This makes important parts of the paper challenging to comprehend, such as the discussion of Eq 9.

Author response:
Despite the complicatedness of the related techniques and the subtleness in planning for POMDPs, which makes the paper technically dense, we try our best to make the paper self-contained, and design a simple algorithm. 
  We have revised our manuscript accordingly and try our best to make the paper more convenient to be understood. We would like to clarify some key typos in the previous manuscript

Reviewer comment (Theory):
The paper is also very dense, which makes some parts too condensed. For instance, the theoretical analysis only states the assumptions and an informal version of the sample complexity of the algorithm without including an analysis of this result.

Author response:
Despite the complicatedness of the related techniques and the subtleness in planning for POMDPs, which makes the paper technically dense, we try our best to make the paper self-contained, and design a simple algorithm. 
  We have revised our manuscript accordingly and try our best to make the paper more convenient to be understood. We would like to clarify some key typos in the previous manuscript

Reviewer comment (Experiments):
The empirical evaluation is limited to a comparison with other algorithms. It would be interesting to provide an ablation study to show how the different components of the algorithm contribute to its performance. For example, how the algorithm performs without optimistic exploration.

Author response:
We have added more ablation studies (Appendix H.1) to demonstrate the effects of the major components, including representation dimension and window size.

Reviewer comment (Writing):
Wrong typesetting of the observation function in the first paragraph of the preliminaries.

Author response:
We would like to clarify some key typos in the previous manuscript

Reviewer comment (Theory):
In the preliminaries, should the agent receive a reward r(s_h, a_h)?

Author response:
In fact the agent can only observe a reward $r(o_h, a_h)$, which can be defined as $$r(o_h, a_h) = \int r(s_h, a_h) P(o_h|s_h) ds_h$$ and can be viewed as a sample from $r(s_h, a_h)$, which is commonly assumed in the existing POMDP literature [1, 2]

Reviewer comment (Writing):
In the belief definition, it is unclear what is P(s1\mid o1). It is also unclear what is \tau.

Author response:
$P(s_1|o_1)$ can be defined as the posterior distribution induced by the initial distribution and emission. $\tau$ is the history defined in the second paragraph of the preliminaries.

Reviewer comment (Writing):
Unclear what is \theta on Eq 5?

Author response:
$\theta$ is just a vector that can induce the reward.

Reviewer comment (Writing):
After Eq 16, could you provide some intuition about what is the parameter l?

Author response:
For any $l \in N_+$, $p({o_{h+1:h+l}}|x_h, a_h) \int_{\mathcal{Z}} p(z_h|x_h, a_h)  p^\pi ({o_{h+1:h+l}}|z_h) d z_h$ Is factorizable and share the same $p(z_h|x_h, a_h)$, but with different  $p^\pi ({o_{h+1:h+l}}|z_h) $. 
  As we discussed that  $p(z_h|x_h, a_h)$ forms the function space for Q-function, and thus, is the latent variable representation we are seeking, we can perform MLE on arbitrary l-step future observations conditional distribution.

Reviewer comment (Writing):
Could you provide a formal definition of the policy \mu_pi?

Author response:
The formal definition of νπ is complicated and we provide a formal definition at the beginning of the proof for Lemma 8. We also add more concrete discussion in Appendix C about the moment matching policy.

Reviewer comment (Theory):
The theoretical analysis relies on quite a few assumptions (including Finite Candidate Class with Realizability, Normalization Conditions, Regularity Conditions and Eigendecay Conditions), which may not always be fulfilled in reality. Can authors comment on the performance of the algorithms when these assumptions break, e.g., how worse the performance is going to be, and which of the assumptions are essential to retain the performance?

Author response:
We first emphasize that the assumptions are merely for the proofs, which are commonly used in the literature (e.g. [1, 2, 3, 4]) and are thought to be mild. While the proposed µLV-Rep is still  widely applicable even without the assumptions in practice. In fact, in our empirical experiments, the assumptions are not guaranteed to be held. But the proposed algorithm still outperforms or is comparable to the existing strong competitors, including DreamerV2, which demonstrates the robustness of the proposed algorithm in practice.

Reviewer comment (Theory):
The theoretical analysis is mainly based on Ren et al., 2023a. It is unclear what are the technical novelties in the analysis compared to Ren et al., 2023a. Authors are expected to explain the difference and highlight the key insights in the proofs. In particular, the proof of Theorem 12 is unclear by just claiming "This is a direct extension of the proof of Theorem 9 in Ren et al. (2023a)". The technical contribution in theory remains questionable.

Author response:
As we mentioned, our core contribution is not theoretical analysis. Instead, our contribution lies on extending LV-Rep for POMDP that can perform provable representation learning and planning with function approximations (as illustrated in Section 4), which is highly nontrivial and significant as we discussed in Section 3, especially considering the research on planning in POMDPs has stuck for decades.
We emphasize the simplicity of the algorithm should be considered as an advantage, which enables the LV-Rep to more practical settings to handle partial observations, rather than a drawback.

Reviewer comment (Novelty):
Algorithmically, the proposed main algorithms borrow lots of the elements from Ren et al., 2023a, and the novelty appears to be limited.

Author response:
As we mentioned, our core contribution is not theoretical analysis. Instead, our contribution lies on extending LV-Rep for POMDP that can perform provable representation learning and planning with function approximations (as illustrated in Section 4), which is highly nontrivial and significant as we discussed in Section 3, especially considering the research on planning in POMDPs has stuck for decades.
We emphasize the simplicity of the algorithm should be considered as an advantage, which enables the LV-Rep to more practical settings to handle partial observations, rather than a drawback.

Reviewer comment (Reproducibility):
In section 4, it is unclear how the planning algorithm implements pessimism for offline RL.

Author response:
The pessimism mechanism in offline setting is similar to the optimism in the online setting, and the major difference between the optimism in online setting and pessimism in offline setting is that, in the offline setting we **minus** the same bonus from the reward while in the online setting we **add** the bonus into the reward. We already added the corresponding discussion in Appendix D our updated version.