Paper: Horizon-Free Regret for Linear Markov Decision Processes (ICLR-2024)

Reviewer comment (Theory):
Providing a slightly clearer explanation of the method in the main text would be beneficial. For instance, as mentioned in Technique 3, calculating an upper bound for \( \max_{h' \in [H]} \sum_{k=1}^K \sum_{h=1}^H \mathbb{V} (P_{s^k_h, a^k_h}, V^*_{h'+1}) \) is required, but it was stated that a bound for \( \sum_{k=1}^K \sum_{i=1}^{\log_2 H +1} \max_{h' \in [H_i]} \sum_{h \in H_i} \mathbb{V} (P_{s^k_h, a^k_h}, V^*_{h+1}) \) was given. A more explicit explanation of how these two bounds are connected would enhance clarity.

Author response:
**About connection between** $\sum_{k=1}^K \sum_{i=1}^{\log_2(H)+1}\max_{h'\in H_i}\sum_{h\in H_i}\mathbb{V}(P_{s_h^k,a_h^k},V_{h'+1}^*)$ and  $\max_{h'\in [H]}\sum_{k=1}^K \sum_{h=1}^H \mathbb{V}(P_{s_h^k,a_h^k},V_{h'+1}^*)$ (Here we use $H_i$ ($V_i$) to replace $\mathcal{H}_i$ $(\mathcal{V}_i)$ so that the Latex editor works):

We try to give a more detailed explanation as following: Our initial target is to bound 
$$\sum_{k=1}^K \min\left\\{  \sum_{h=1}^H \sqrt{  (\phi_h^k)^{\top}  (\Lambda^k(V_{h+1}^*))^{-1}\phi_h^k  }  ,1\right\\}.$$

Following the naive maximal variance approach, we need to bound the later term  $\max_{h'\in [H]}\sum_{k=1}^K \sum_{h=1}^H \mathbb{V}(P_{s_h^k,a_h^k},V_{h'+1}^*)$, which is still hard (as presented by the counter example). 
Instead, we turn to bound $$\max_{h'\in [H]}\sum_{k=1}^K \sum_{h=1}^H \mathbb{V}(P_{s_h^k,a_h^k},V_{h'+1}^*)\leq \sum_{i=1}^{\log_2(H)+1}\sum_{k=1}^K \min\left\\{ \sum_{h\in H_i}\sqrt{  (\phi_h^k)^{\top}(\Lambda^k(V_{h+1}^*))^{-1}\phi_h^k } ,1\right\\},$$ 
where we could use the bound for the first term  $\sum_{k=1}^K \sum_{i=1}^{\log_2(H)+1}\max_{h'\in H_i}\sum_{h\in H_i}\mathbb{V}(P_{s_h^k,a_h^k},V_{h'+1}^*)$.

Reviewer comment (Theory):
On page 7, could you please provide an explanation for the statement, "\( \mathbb{V}(P_{s,a}, v) = \phi(s,a)^\top (\theta(v^2)) - (\phi(s,a)^\top \theta(v))^2 \)" being described as a linear function of the matrix?

Author response:
**About the statement** "$\mathbb{V}(P_{s,a},v)= \phi(s,a)^{\top} (\theta(v^2)) - (\phi(s,a)^{\top}\theta(v))^2$" in page 7:

We can write $\mathbb{V}(P_{s,a},v)= \phi(s,a)^{\top} (\theta(v^2)) - (\phi(s,a)^{\top}\theta(v))^2$ as $\mathrm{Trace}(L^{\top}(s,a) B(v))$ where
$$
L(s,a)=
\left[\begin{array}{cc}
  \phi(s,a)\phi^{\top}(s,a)  , &  \phi(s,a),\\\\
   \phi^{\top}(s,a), & 1
\end{array}\right] $$ 
and
$$
B(v)= \left[\begin{array}{cc}
  -\theta(v)\theta^{\top}(v) ,  &   \theta(v^2)/2,\\\\
  \theta^{\top}(v^2)/2 ,& 0
\end{array}\right]
 $$
For fixed $(s,a)$, $\mathbb{V}(P_{s,a},v)$ is a linear function of the matrix $B(v)$. In Equation.(42), with Lemma 5 we derive that
\begin{align}\sum_{k=1}^K\sum_{h\in H_i}\max_{v\in V_i}\mathbb{V}(P_{s_h^k,a_h^k},v) & = \sum_{k=1}^K\sum_{h\in H_i}\max_{v\in V_i}\mathrm{Trace}(L^{\top}(s_h^k,a_h^k)B(v)) 
\\\\ & \leq 2(d+1)^2\max_{v\in V_i}\sum_{k=1}^K\sum_{h\in H_i}\mathrm{Trace}(L^{\top}(s_h^k,a_h^k)B(v))\\\\&=2(d+1)^2\max_{v\in V_i}\sum_{k=1}^K\sum_{h\in H_i}\mathbb{V}(P_{s_h^k,a_h^k},v).
\end{align}

Reviewer comment (Theory):
Could you clarify how the last inequality in equation (11) holds?

Author response:
**About the last inequality in Eq.(11):**

We have that 
$$ \max_{\phi\in\Phi} 2d\phi^{\top}\sum_{h=1}^{H-1}\mu^{\top} (V_{h+1}-V_{h+2})  =2d \max_{\phi\in \Phi}\phi^{\top}\mu^{\top}V_{2} =2d\max_{s,a}\phi_{s,a}^{\top}\mu^{\top}V_2.$$
Continuing the computation, we have
$$ \max_{\phi\in\Phi} 2d\phi^{\top}\sum_{h=1}^{H-1}\mu^{\top} (V_{h+1}-V_{h+2}) =2d\max_{s,a}P_{s,a}^{\top}V_{2}\leq 2d\max_{s}V_1(s)\leq 2d.$$

Reviewer comment (Writing):
Typo?: It seems that while \( l_h^* \) is defined below equation (10), \( l_h \) is not defined. Please provide a definition for \( l_h \).

Author response:
**About $l_h^{*}$:**

Thank you for point this out. It should be $l_h$ but not $l_h^*$.

Reviewer comment (Writing):
What is the meaning of "size of transition model"?

Author response:
**About “size of transition model”:** 

It means the number of parameters needed to fully describe the transition model.

Reviewer comment (Presentation):
The structure of the paper can be improved. It seems the structure follows Zhou & Gu (2022), but the demonstration of "technical challenge for 3.5 pages without demonstrating algorithm (the algorithm is introduced after that) does not make much sense. At least, this 3.5 pages does not help the reader to be convinced of the results more than 1-page summary.

Author response:
**About paper structure:** 
We have adjusted the structure of this paper and presented the algorithm before introducing the techniques.

Reviewer comment (Writing):
There are many typos, which makes it hard for the reader to be convinced of the results.

Author response:
**About the typos:** 
We have fixed the typos accordingly in the revision. Thanks for the effort to check the correctness of this work.

Reviewer comment (Writing):
My largest question is about Algorithm 1. If I understand correctly, these algorithms require optimization over continuous space. First, $\mathcal{W}(\epsilon)$ is not defined in this paper; similar quantities such as $\mathcal{W}$ and $\mathcal{W}_\epsilon$ are defined twice, so I assume these are equivalent. Lines 6--9 in Alg1 loops over $v \in \mathcal{W}(\epsilon)$, which defines quantities $(\hat{\theta}, \tilde{\theta}, \Lambda^k(v), b^k(v, \phi))$ as a function of $v$. $\mathcal{U}^k$ is also optimized on the space of $v$.

Author response:
**About Algorithm 1:**

**About $\mathcal{W}(\epsilon)$:**

$\mathcal{W}$ is  a continuous set and $\mathcal{W}_{\epsilon}$ is the discretization of $\mathcal{W}$. 

You are correct that $\mathcal{W}(\epsilon)$ and $\mathcal{W}_{\epsilon}$ denote the same set.  

We replaced $\mathcal{W}(\epsilon)$ by $\mathcal{W}_{\epsilon}$ and updated the paper accordingly. Thanks for pointing out!

Reviewer comment (Presentation):
$\tilde{\theta}(v)$ is not used in the line 10 of Alg 1 and after.

Author response:
**$\tilde{\theta}(v)$ is not used in the line 10 of Alg 1 and after?** 

We do not use $\tilde{\theta}(v)$ in the main algorithm.  $\tilde{\theta}$ in the output of Algorithm 3 is used in the statement and proof of Lemma 10, which states that $(\hat{\theta}, \tilde{\theta})$ is a close estimation of $(\theta(v),\theta(v^2) )$.

Reviewer comment (Theory):
On the similarity with the (contextual) linear bandit problem. The linear bandit problem is where $x$ is disclosed (contextual is $x$ is $x(t)$) and reward is $r(t) = x^\top \theta$ where $\theta$ is an unknown quantity. In the MDP of this paper, transition has uncertainty and the uncertainty on the reward $r(t)$ w.r.t. transition kernel is much more involved.

Author response:
**Similarity with contextual linear bandit problem:**

We believe linear MDP has the same intrinsic hardness as contextual linear bandit problem. The transitions are involved but it suffices to learn $\theta^*_h$  for $h \in [H]$ to make decisions.

Reviewer comment (Theory):
VOFUL: Is it by any means related to OFUL? If I understand correctly, OFUL (e.g., Abbasi-Yadkori et al. 2011) is an algorithm for online bandit algorithm whereas Alg 3 is for the uniform confidence bound (also referred as self-normalized bound or martingale bound).

Author response:
**About VOFUL:**

VOFUL is an algorithm with variance-awared regret bound for linear bandits. The high-level idea is to construct a confidence region for the unknown parameter $\theta$ and then choose arm optimistically. Both VOFUL and OFUL rely on optimism but the algorithm structures are quite different.

Reviewer comment (Writing):
Typos. Many periods are missing, for examples:  
> p2: See Section 3 for more details. Due to space limitation, we postpone the full proof of Theorem 1 to Appendix B  
> p6: That is, we need to use all the samples along the trajectory...  
> p7: Fix some eps>0

Author response:
**About the typos:** 
We have fixed the typos accordingly in the revision. Thanks for the effort to check the correctness of this work.

Reviewer comment (Theory):
Why it holds \( f(l-y)/y^{d-1} \le f(l-x)/x^{d-1} \)?

Author response:
#### Why $\frac{f(l-y)}{y^{d-1}}\leq \frac{f(l-x)}{x^{d-1}}$:  
We have added a detailed proof in the appendix page 13 (marked in red).  
At a high-level, $f(x)$ denote the $d-1$-dimensional volume of the slice $\mathcal{V}(x):=\Phi \cap \{\phi_1  = x | \phi\in \mathbb{R}^d \}$ (here $\phi_1$ denotes the first dimension of $\phi$). Note that $\mathcal{V}(l)$ is non-empty, and $\Phi$ is convex and closed. Let $\tilde{\phi}$ be an arbitrary element of $\mathcal{V}(l)$. We then have that $\frac{x}{y}\mathcal{V}(l-y) + \frac{y-x}{y}\tilde{\phi}:= \{ \frac{x}{y}\phi+\frac{y-x}{y}\tilde{\phi}｜ \phi\in \mathcal{V}(l-y) \}\subset \mathcal{V}(l-x)$, which implies the $d-1$-dimensional volume of $\mathcal{V}(l-x)$ is at least $(\frac{x}{y})^{d-1}$ fraction of that of $\mathcal{V}(l-y)$ for $0<x\leq y \leq l$.

Reviewer comment (Theory):
Is there any references that will show why equation (16) holds?

Author response:
**About (16):**  

After correcting the definition $\bar{\phi}$, 
\begin{align}
z &= \frac{\int_{\Phi}\phi^{\top}\psi du(\phi)}{\int_{\phi}du(\phi)} \nonumber
\\ & = \lim_{\epsilon\to 0}\sum_{i=0}^{\left\lceil l/\epsilon\right\rceil}\frac{\int_{\phi\in \Phi, \phi^{\top}\psi \in [i\epsilon,(i+1)\epsilon) } \phi^{\top}\psi du(\phi) }{\int_{\Phi}du(\phi)}\nonumber
\\ &  = \lim_{\epsilon\to 0}\sum_{i=0}^{\left\lceil l/\epsilon\right\rceil}\frac{\int_{\phi\in \Phi, \phi^{\top}\psi \in [i\epsilon,(i+1)\epsilon) } i\epsilon du(\phi) }{\int_{\Phi}du(\phi)}\nonumber
\\ & = \lim_{\epsilon\to 0}\sum_{i=0}^{\left\lceil l/\epsilon\right\rceil}f(i\epsilon)i\epsilon^2\nonumber
\\ & = \int_{0}^l f(x)xdx. \nonumber
\end{align}

By re-arrange the inequality, and noting that $int_{0}^{l}f(x)dx = 1$ we have that 
\begin{align}
\int_{ 0\leq x \leq z}f(x)\cdot (z-x) d(x)= \int_{ z\leq x\leq l} f(x)(x-z)dx, \nonumber
\end{align}

Reviewer comment (Writing):
Definition of \( \bar v(s) \) in Lemma 11 is not clearly written: is it \( \bar v(s) = \max_{a} P_{s,a} v \)?

Author response:
**About $\bar{v}(s)$:**  
It should be $\bar{v}(s) = \max\{ \max_a P_{s,a}v, v(s) \}$.

Reviewer comment (Theory):
It is not clear in Lemma 11 why (c) holds because it is not easily understandable why the presented \( \bar v \) maximizes \( v(s^k_{h+1}) \), or I did not understand the definition of \( \bar v \).

Author response:
**About Equation \(c\).** with the definition of $\bar{v}(s)$ above, we have 
$$\max\{ v(s_h^k)-P_{s,a}v, 0 \} \leq  \max\{ \bar{v}(s_h^k)-P_{s,a}v,0 \} =\bar{v}(s_h^k)-P_{s,a}v.  $$

Reviewer comment (Theory):
Equations (9) and (13) — missed square in \( \bar \sigma^k_h \) in the second multiplier after Cauchy–Schwartz inequality.

Author response:
**Typos:**  

**About $\sigma$:** we use $\sigma^2$ to denote the variance term throughout the paper. We fix the typos in the revision.

Reviewer comment (Theory):
After equation (9) it claims that \( \sigma^k_h \) is variance, whereas before \( \sigma^k_h \) was a square root of variance; this is very confusing.

Author response:
**Typos:**  

**About $\sigma$:** we use $\sigma^2$ to denote the variance term throughout the paper. We fix the typos in the revision.

Reviewer comment (Writing):
Proof of Lemma 5: \( O \) is not defined.

Author response:
**About $O$ in the proof of Lemma 5.** $O$ should be $\max_j \sum_{i=1}^n \phi_i^{\top}\psi_j$.  We fix this inequality and delete $O$.

Reviewer comment (Writing):
After equation (28): use of Var as variance whereas usually it was denoted by \( \mathbb{V} \), and after it there is a confusing claim \( \sigma^k_h(v) \geq \text{Var} \).

Author response:
**About $Var$ in Eq.(28):** we replace it by $\mathbb{V}$.

Reviewer comment (Writing):
After equation (29) — no brackets around \( i \) in the inequality \( \bar \Lambda^k_i \leq \Lambda^k(\bar V^k_{h+1}) \).

Author response:
**About Eq.(29):** We add the bracket to $i$.

Reviewer comment (Theory):
After equation (30) — forgotten subscript \((i)\) in the definition of \( \bar \Lambda^k \) and it is not clear why \( \bar \Lambda^k + \sum_{h=1}^H \dots = \bar \Lambda^{k+1} \).

Author response:
**About Eq.(30):** We replace $\bar{\Lambda}^k$ by $\bar{\Lambda}^k_{(i)}$ in the analysis after Eq.(30). 
 The equation 
$$\bar{\Lambda}^k_{(i)}+\sum_{h\in H_i}\frac{\phi_{h}^{k}(\phi_{h}^{k})^{\top}}{\max_{v\in V_i}(\sigma_{h}^{k}(v))^2} \preccurlyeq \bar{\Lambda}^k_{(i)} + \sum_{h=1}^H\frac{\phi_{h}^{k} (\phi_{h}^{k})^{\top}}{\max_{v\in V_i}(\sigma_{h}^{k}(v))^2} =\bar{\Lambda}^{k+1}_{(i)}$$

should be replaced by 
$$\bar{\Lambda}^k_{(i)} + \sum_{h\in H_{i}}\frac{\phi_{h}^{k} (\phi_{h}^{k})^{\top}}{\max_{v\in V_i}(\sigma_{h}^{k}(v))^2}  =\bar{\Lambda}^{k+1}_{(i)}$$

Reviewer comment (Writing):
Last paragraph of page 21: undefined norm type \( \Vert \bar v - v \Vert_i \).

Author response:
**About $\|\bar{v}-v\|_i$:** it should be $\|\bar{v}-v\|_{\infty}$.

Reviewer comment (Theory):
Lemma 12 — missed norm type in the statement (is it \( \ell_2 \) norm or \( \ell_\infty \) as used later?).

Author response:
**About norm type in Lemma 12:** it should be $\infty$ norm