Paper: Provably Efficient Exploration in Quantum Reinforcement Learning with Logarithmic Worst-Case Regret (ICLR-2024)

Reviewer comment (Novelty):
Lack of novelty. I am familiar with the work of Wan et al 2022 for quantum multi-armed bandits and quantum linear bandits. As the main theoretical tools for multi-armed bandits and linear bandits are considerably similar to RL and linear RL respectively, this paper can be regarded as an extension from Wan et al 2022 to quantum RL. Although the author pointed out one new challenge in Remark 5.1, I did not see enough novel contributions in this work.

Author response:
We appreciate the thoughtful review and have carefully considered the concerns raised. We would like to address the primary issue of perceived lack of novelty by providing a detailed clarification of the unique contributions of our work.

**Regarding lack of novelty.**  Our work acknowledges its relationship with [1], extensively discussed in the related work and main paper, particularly in Remark 5.1. While it is true that multi-armed bandits and linear bandits are special cases of tabular and linear mixture MDPs, respectively, learning MDPs with multi-step decision-making introduces substantial complexity beyond bandit scenarios. We highlight the technical difficulties and our novel contributions. 


**Algorithm design novelties:**
- One can regard the bandits as the MDPs satisfying (i) the state space only contains a single dummy state $s_{\mathrm{dummy}}$; (ii) $H=1$; and (iii) the reward only depends on the action chosen by the learner, i.e., $r(s_{\mathrm{dummy}}, a) = r(a)$. In this case, the learner can repeatedly pull a particular arm $a$ to collect sufficient samples to estimate $r(s_{\mathrm{dummy}}, a)$. However, in online RL, the state will transit according to the transition kernel, preventing repeated arm pulls for the desired estimator. To address this, we introduce a novel adaptive lazy-updating rule, quantifying the uncertainty of the visiting state-action pair (covariance matix in linear setting) and updating the model infrequently through the doubling trick. **This algorithm design, absent in previous works on quantum bandits [1] and quantum RL with a generative model [2], connects online exploration in quantum RL with strategic lazy updating mechanisms, inspiring subsequent algorithm design in online quantum RL.**


**Technical novelties:**
 - In the **tabular** setting, a significant technical challenge is obtaining a high probability regret bound, as vanilla regret decomposition (e.g., the one used in UCBVI) leads to a martingale term of order $\mathcal{O}(H\sqrt{T})$. Though it is not the dominating term in the classical setting, it would become the dominating term in our setting. **We found that another regret decomposition (i.e., the one used in EULER) based on the expected Bellman error cleverly bypasses such martingale terms in the regret, thus achieving the desired regret bound.**
 - In the **linear** setting, the algorithm requires a novel design to leverage the advantage of quantum mean estimation. Classical linear RL analysis heavily relies on martingale concentration, such as the well-known self-normalized concentration bound for vectors. However, in quantum RL, there is no direct counterpart to martingale analysis. Consequently, **we redesign the lazy updating frequency and propose an entirely new technique to estimate features for building the confidence set for the model.** Notably, previous works on quantum bandits [1] do not need to take this step since there is no unknown transition kernel in their setting. Moreover, estimating the features poses a **subtle technical challenge, as elaborated in Equation (5.5).** Our proposed algorithm (Algorithm 2), which features **binary search and quantum mean estimation**, successfully addresses this technical challenge. Meanwhile, **the quantum samples used in feature estimation approximately equal the number of quantum samples in each phase, eliminating extra regret for this additional feature estimation.** This algorithm design (Algortihm 2) and theoretical analysis are **entirely new** in the literature on (classical and quantum) RL theory. We sincerely hope the reviewer can reconsider the novelty of this part. We believe that **introducing a new algorithm design and novel techniques with interesting results is sufficient for the first theoretical study in online quantum RL (with function approximation).**

**Significance of our results:**
- Beyond our novelties in algorithm design and techniques, the paramount significance lies in the message that quantum computing can potentially provide a speedup for online sequential decision-making problems under the framework of MDPs. The logarithmic regret obtained is markedly different from classical $\sqrt{T}$ regret, providing a theoretical guarantee for the potential superiority of quantum RL over classical RL. As **the first rigorously theoretical work on online quantum RL**, our research is deemed significant, paving the way for future investigations in the realm of quantum RL.


According to your concerns, we have made a thorough revision of our description of novelties. Please refer to the **introduction section, Remark 5.1, the proof sketch of Theorem 5.2, and Appendix B.2 for details, as highlighted in blue.**

Reviewer comment (Novelty):
Although leaning toward a negative evaluation of this work for its lack of contribution, I think this quantum RL topic is interesting and would suggest that the authors look into challenging issues around this topic, e.g., regret lower bounds for quantum RL which is not studied in quantum bandits in Wan et al 2022.

Author response:
**Regarding regret lower bound.** Please see our general response for details.

Reviewer comment (Theory):
The discussion on sample complexity is not enough. For example, it would be better to discuss why the $\sqrt{T}$ factor is removed. Is that because of Lemma 3.1 and Lemma 3.2 such that the previous $\epsilon^{-2}$ sample complexity can be reduced to $\epsilon^{-1}$ sample complexity so that the exploration can be more aggressive?

Author response:
Yes, the logarithmic regret in our work is primarily attributed to the speedup of quantum mean estimation (Lemma 3.1 and Lemma 3.2). Simultaneously, the $\epsilon^{-1}$ sample complexity leads to more conservative exploration instead of aggressive, as we can make more accurate estimations. This conservatism is further reflected in the bonus term; our bonus term $1/n_h^k(s, a)$ is smaller than the classical $1/\sqrt{n_h^k(s, a)}$, resulting in a more cautious exploration strategy. We appreciate your valuable insights and comments, and we have incorporated these discussions into the introduction, highlighted in blue.

Moreover, the strategic combination of the speedup in quantum mean estimations and exploration in online quantum RL is the central focus of our algorithm design and analysis. Please refer to our introduction section and other parts of the main paper for a detailed elaboration of this aspect.

Reviewer comment (Theory):
Can one directly covert the observation in classical RL to a quantum-accessible RL? (e.g., changing the Atari games to quantum). If the quantum RL can be used in classical RL tasks, then how would the current $\log T$ bound break the classical $\sqrt{T}$ regret bound?

Author response:
We can run classical RL on quantum computers as a special case. For instance, we can play classical Atari games on quantum computers when they become universal and have enough number of qubits to execute the programs. That is, quantum computers can simulate classical RL environments. Moreover, quantum accessible environments empower the agent to hold the quantum superpositions over states and actions, which encode the stochasity of the envrionments. In contrast, classical environment only returns random samples from the environment.

Reviewer comment (Novelty):
If the current algorithm can only be used in quantum-accessible RL, and we cannot convert a classical RL task into quantum, then how will this algorithm contribute to real-world RL tasks?

Author response:
Same as A3, we can regard a classical RL task as a quantum RL task but never uses superposition. The algorithm proposed in this paper is a quantum RL algorithm and can only be executed on quantum computers. In terms of the contribution to real-world RL tasks, due to the limitation of current quantum computers we have to focus on theoretical research of quantum RL. Nevertheless, our result is fundamental as we proved the first poly-log regret results for quantum RL, and we believe that our work will have real-world impact after the capability of quantum computers significantly grows.

Reviewer comment (Experiments):
This work does not have any empirical study of the proposed algorithms.

Author response:
**A7**: While we appreciate the suggestion for an experimental study, our primary focus is on advancing the theoretical understanding of quantum reinforcement learning (RL). Given current limitations in quantum computing capabilities, our strategic emphasis on theory aims to lay a foundational framework. We believe that this theoretical groundwork will inspire both theoretical and empirical advancements in quantum RL as quantum computing technology evolves.

Reviewer comment (Theory):
Could the authors comment on why the binary oracle is not considered in the tabular setting? What is the main difficulty in generalizing the result of Lemma 3.1 to binary oracle?

Author response:
**A4**: We can definitely generalize the results of Lemma 3.1 to Lemma 3.2 with binary oracle in the tabular setting. This is done in the following way. For a fixed $(s, a)$ pair, we define the binary oracle $U_{s, a} : \ket{s, a} \ket{s'} \ket{0} \to \ket{s, a} \ket{\vec{1}[s']}$, where $\vec{1}[s']$ is an $S$-dimensional standard basis vector encoded by $s'$ (i.e., a one-hot vector with non-zero entry at the $s'$ position). Combined with the probability oracle $\bar{\mathcal{P}}\_h: \ket{s, a} \ket{0} \to \sum_{s'} \sqrt{\mathcal{P}\_h(s' \mid s, a)} \ket{s'}$, we can estimate the $S$-dimensional vector $(\mathcal{P}\_h(s_1 \mid s, a), \mathcal{P}\_h(s_2 \mid s, a), ..., \mathcal{P}\_h(s_{S} \mid s, a))^\top \in \mathbb{R}^S$ suppose $s'$ is enumerated from the state space $\{s_1, s_2, ..., s_{S}\}$ **using Lemma 3.2**. Since we hope the $l_1$ estimation error to be bouned by $\epsilon$, the sample complexity of such estimation is $\mathcal{O}(S \log (S/\delta) / \epsilon)$, the same as Lemma 3.1. Since the target vector $(\mathcal{P}\_h(s_1 \mid s, a), \mathcal{P}\_h(s_2 \mid s, a), ..., \mathcal{P}\_h(s_{S} \mid s, a))$ is actually a distribution over the state space, we can also encode this vector as the amplitude of a quantum superposition and use Lemma 3.1 to estimate the amplitude. *We choose Lemma 3.1 mainly because the quantum multi-dimensional amplitude estimation subroutine is conceptually simpler without requiring an additional binary oracle, so it is more efficient in implementation.*

Reviewer comment (Writing):
The introduction/description of the Quantum UCRL algorithm is not clear enough. Specifically, I could not find the $\bar{\varphi}_{h+1}, \mathcal{D}_h(s^k_h, a^k_h)$ notations appeared in Algorithm 4 being defined anywhere.

Author response:
**A2**: We greatly appreciate the reviewer's suggestion to clarify the confusing notations/definitions in our paper. To be specific, the place $\bar{\varphi}\_{h+1}$ appeared in Algorithm 4 (i.e., Line 7 of Algorithm 4) is exactly the definition of $\bar{\varphi}\_{h+1}$. According to the context, $\bar{\varphi}\_{h+1}$ is **defined as** the output of the oracle $\bar{\mathcal{P}}\_h$ (defined by Eqn. (3.2)) queried on input $\left|s_h^k, a_h^k\right\rangle|0\rangle$. We should use $\ket{\bar{\varphi}\_{k, h+1}}$ to represent this output because it is a quantum superposition obtained in episode $k$. Similarly, the definition of $\bar{\varphi}\_{h+1}^{-1}$ is $\bar{\varphi}\_{h+1}^{-1} := \bar{\mathcal{P}}\_h^{-1} \left|s_h^k, a_h^k\right\rangle|0\rangle$ (we will also use $\ket{\bar{\varphi}\_{k,h+1}^{-1}}$ instead of $\bar{\varphi}\_{h+1}^{-1}$ in the revised version of the paper). 

The notation $\mathcal{D}_h(s_h^k, a_h^k)$ is defined at Line 2 of Algorithm 4 (the third bullet point). It is defined as the collection (i.e., the multiset) of all the quantum samples obtained for the state-action pair $(s_h^k, a_h^k)$ at step $h$. We will use multiset $\mathcal{D}_h(s_h^k, a_h^k)$ as input to call the quantum multi-dimensional amplitude estimation algorithm (Lemma 3.1).

Reviewer comment (Theory):
If $\bar{\varphi}_{h+1}$ is as defined at the end of subsection 3.2, then it is a quantum state in superposition. How could Algorithm 4 line 9 update the counter according to the superposition? Please correct me if I missed anything.

Author response:
**A3**: According to the definition, $\bar{\varphi}\_{h+1} = \bar{\mathcal{P}}\_{h} \ket{s_{h}^k, a_{h}^k} \ket{0}$ is indeed a quantum superposition, with which we cannot update the counter. Fortunately, $(s_h^k, a_h^k)$ is a classical state-action pair instead of a quantum superposition, and $\ket{s_h^k, a_h^k}$ is a standard basis vector of Hilbert space $\bar{S} \times \bar{A}$ (defined at the beginning of the second paragraph of Section 3.2) encoded by $(s_h^k, a_h^k)$. This is because $s_h^k$ is the output of Algorithm 3, which always outputs a classical state sampled from some distribution. $a_h^k$ is also a classical action since $a_h^k := \pi_h^k(s_h^k)$. Therefore, we can update the counter according to $(s_h^k, a_h^k)$. Moreover, this is a key reason why we need to design a complicated phase-based "sampling" procedure in Algorithm 4 to obtain valid quantum samples, which is a dramatic difference between quantum RL and classical RL.

Reviewer comment (Reproducibility):
Why does Quantum UCRL divide the episodes into T/H phases while Quantum UCRL-VTR divides into K phases? How should the practitioners set the parameter K for Quantum UCRL-VTR?

Author response:
**A5**: Both the Quantum UCRL and Quantum UCLR-VTR algorithms leverage the idea of lazy updating: we divide the total $T$ episodes into $K (K \leq T)$ phases (each phase contains some contiguous episodes), and update the estimation of the parameter **only once in a single phase**. The purpose of lazy updating is to use quantum samples efficiently, since any quantum sample can be used only once (after then it collapses). The tabular setting is easier, in which we can use schedule the length of each phase to $H$ equally. The linear setting is more complicated, as we need to schedule the length of each phase **on the fly**: we decide the length of next phase at the end of last phase (the length of the $k$-th phase is $\tilde{\mathcal{O}}(\sqrt{d}H^2 / w_k)$ where $w\_k := \max\_{h \in[H]}||\widehat{\phi}\_h^k||\_{\left(\Lambda_h^k\right)^{-1}}$ according to Line 6 of Algorithm 1). It is guaranteed that the total number of phases will be bouned by $\mathcal{O}(dH \log T)$ according to Lemma F.1. Therefore, we do not need to choose the parameter $K$ at the beginning of Quantum UCRL-VTR. It suffices to follow the decision rule of the next phase length of Quantum UCRL-VTR. **We have added a proof sketch of Theorem 5.2 (see the second paragraph below Theorem 5.2, highlighted in blue), where we clarify the size of the phase number $K$ and the number of episodes in the $k$-th phase. Thank you for your question!**

Reviewer comment (Theory):
The quantum oracle for reward function is not used as it is assumed to be known for the problems considered in the paper. Is this right?

Author response:
**A5:** Yes, our quantum RL algorithms do not utilize the quantum oracle for the reward function under the assumption that the reward function is known. This assumption is mild and widely adopted in the RL theory literature.  It serves to simplify the presentation and theoretical analysis, yet retains its significance by addressing the most challenging aspect of RL --- learning unknown transition kernels. Furthermore, our results extend seamlessly to the unknown reward setting. In this scenario, we utilize quantum estimation oracles (Lemma 3.2) to estimate the unknown reward function. Notably, in the unknown reward setting, our final regret bounds remain consistent with those in the known reward setting (our current results). This alignment arises from the fact that the additional reward estimation error, roughly $1/n_h^k(s, a)$, is smaller than the transition estimation error, roughly $S/n_h^k(s,a)$. This reaffirms the notion that learning transition kernels poses a greater difficulty than learning the reward function, thereby lending support to the rationality of the known reward assumption.

Reviewer comment (Experiments):
If possible, authors can add a few experiments using the Python library QisKit. It will make the paper stronger.

Author response:
**A6:** Thank you for your suggestions. We want to emphasize that our work places a primary emphasis on the theoretical aspects of online quantum RL, with a central goal of providing the first logarithmic regret. We firmly believe that our theoretical results stand as a significant contribution in their own right and merit acceptance. Experiments are though interesting questions and can be left for future works.