Paper: Model-Free, Regret-Optimal Best Policy Identification in Online CMDPs (ICLR-2024)

Reviewer comment (Theory):
Some assumptions of this paper seem a bit strong. In particular, this paper requires that each state-action pair can be visited with nontrivial probability $p_{min}$. Also, Assumptions 2 and 5 seem unnatural to me. When comparing with existing works, it would be nice to also compare with the assumptions made in these works.

Author response:
**Weakness 1 and Q1.** Assumption 2 and 5 means when the policy deviates from the optimal policy, then either the reward value function or the utility value function, or both should also deviate from those under the optimal policy.  We believe this is a reasonable assumption and should hold for most CMDPs.

Reviewer comment (Experiments):
It would be great if the authors could conduct some simulation experiments that shows the the optimal policy is found, not just regret and constraint violation.

Author response:
**Weakness 3.** For the synthetic CMDP simulation result, we indeed show that the algorithm finds a policy with cumulative reward 1.57301 and cumulative utility 2.00008. The cumulative reward and cumulative utility under the optimal solution are 1.57306 and 2, respectively. This comparison is given at the beginning of page 9. We will further highlight it in the revision.

Reviewer comment (Theory):
In Theorem 4, do you get exactly an optimal policy or an approximate one as in Theorem 3.

Author response:
**Q3.** The algorithm outputs a near-optimal solution. We will clarify this in the revision.

Reviewer comment (Theory):
Can you replace Triple-Q with any efficient CMDP algorithm as long as the regret and constraint violation are sublinear? Suppose the regret is $K^{\alpha}$ and constraint violation is $K^{\beta}$. What would be the error in learning the policy? I am asking because it seems that the proof of Theorem 3 only requires some regret and constraint violation results.

Author response:
**Weakness 2 and Q4.** As the reviewer correctly pointed out that the results of this paper hold when we replace Triple-Q with any online, model-free CMDP algorithm with sublinear regret and constraint violation.  PRI can be viewed as a ``meta-algorithm'' that builds on any model-free CMDP algorithm with sublinear regret and constraint violation to achieve  $\tilde{O}(\sqrt{K})$ regret and constraint violation and to output a single stochastic policy.

Reviewer comment (Novelty):
Episodic CMDP can be considered a special case of infinite-horizon average reward CMDP. An average reward CMDP can be transformed into an episodic CMDP by substituting $T=HK$ and augmenting a time index modulo $H$ in the state description. Therefore, Table 1 should also include the equivalent results obtained from the average reward CMDP literature.

Author response:
**Q1.** Thanks for the suggestion! We will add the discussion on its connection to average reward CMDPs.

Reviewer comment (Writing):
The term $M$ in Lemma 2 has not been explicitly defined.

Author response:
**Q3.** $M$ is the same as the $M_q$ defined on page 4. We will clarify this in the revision.

Reviewer comment (Reproducibility):
There should be a policy initialization in Algorithm 1.

Author response:
**Q4.** Sorry for the confusion. The policy initialization is the same as that of Triple-Q. We will clarify this in the revision.

Reviewer comment (Writing):
Please use a different notation for the coefficients in $(11)$. The current one is similar to the notation of an action.

Author response:
**Q5.** Good point! We will change it to $\alpha_m$ in the revision.

Reviewer comment (Theory):
It seems that the number of optimization variables in $(11)$ is $\mathcal{O}(A^{SH})$ which could make the problem prohibitive for a large state space. This should be clearly stated in the paper.

Author response:
**Q6.** The number of optimization variables is $M,$ the number of greedy policies after the pruning phase. $M$ is upper bounded by $2^N,$ where $N$ is the number of constraints. We will clarify this in the revision.

Reviewer comment (Theory):
It seems that Assumption 2 is redundant for finite state and action spaces.

Author response:
**Q8.** Can you please elaborate? It is not clear to us if the assumption automatically holds for finite state and action spaces.

Reviewer comment (Theory):
The policy identification stage is run for $\mathcal{O}(MKH)$ number of steps and as stated before, $M$, in the worst-case can be $\mathcal{O}(A^{SH})$. Why this bound does not appear in the final regret should be intuitively explained.

Author response:
**Q9.** $M$ is bounded by $2^N,$ where $N$ is the number of constraints. Also, $\sum_{i=1}^M a_i = 1$ so the policy identification stage includes $K$ episodes.

Reviewer comment (Theory):
Theorem 3 dictates the BPI result assuming perfect pruning which does not happen with at most $\mathcal{O}(K^{-0.1})$ probability. In the introduction, this probability is mentioned to be $\mathcal{O}(1/\sqrt{K})$. Please clarify.

Author response:
**Q10.** Sorry for the typo. It should be $O(K^{-0.1})$ in the introduction as well.

Reviewer comment (Theory):
Should Assumption 3 hold for any greedy policy $\pi$? It looks a little strong in its current form.

Author response:
**Q2**. Yes, it needs to hold for any greedy policy. We later relax this assumption when we add the multi-solution pruning phase.

Reviewer comment (Writing):
My only suggestion for improving the clarity is to add a short review of Triple-Q to make the paper more self-contained.

Author response:
**Weakness**. Thank you very much for the suggestion! We will add the review of Triple-Q in the revision.

Reviewer comment (Writing):
In equation (3), does $\rho^n$ mean the exponential of $\rho$, or just a constant that differs with $n$? If it is an exponential of $\rho$, can the authors explain why considering this special form? What's the difficulty of considering general $\rho_n$? If it is indeed a constant that takes different values with $n$, then I suggest using $\rho_n$ to avoid confusion.

Author response:
**Q1**. It is a constant a constant for constraint $n,$ not the exponential. We will use $(n)$ in the revision to avoid this confusion. This paper the subscripts are related to time (step index, episode index, etc) so we feel it is better to superscript as the constraint index.

Reviewer comment (Theory):
In Algorithm 1, what if K is unknown? How to implement the algorithm? Does Theorem 1-3 still hold?

Author response:
**Q2**. When $K$ is unknown, we may use the standard doubling trick, i.e., running the algorithm by increasing the assumed horizon from $T$, $2T,$ to $4T,$ $\cdots$ until the process ends.

Reviewer comment (Evaluation):
In section 7, why does Triple Q have a much smaller negative constraint violation? Is it because Triple Q becomes very conservative in the end? But shouldn't the conservativeness reduce as learning continues? Besides, instead of total constraint violation, what's the total number of episodes or stages of constraint violation?

Author response:
**Q3**. Yes, the reviewer is correct that Triple-Q is conservative and aims at achieving zero constraint violation, so it has smaller negative constraint violation. While the conservativeness reduces under Triple-Q, the plot shows the total constraint violation so it continues to decreases. We did not count the number of episodes where the constraint is violated because the constraint violation is defined on the utility-value function, which is the expected total utility, averaged over many episodes.

Reviewer comment (Theory):
Since the paper refers to deterministic CMDP (and even assuming the generalisation to stochastic rewards and constraints to be trivial), the notion of violation proposed seems to be weak. Indeed, [Efroni et al., 2020] model-based methods, achieves optimal sublinear violation when the cancellations between episodes are not possible.

Author response:
We thank the reviewer for this comment. While the constraint violation defined in the paper allows the cancellation across episodes, we can guarantee $O(\sqrt{K}\log K)$ violation when the cancellation is not allowed by making some minor modification to the algorithm. In particular, in the policy refinement and identification phases, instead of using the $M$ greedy policy in a round-robin fashion, we can use a mixed policy that chooses policy $m$ with probability $a_m.$ With that modification, we will have
* The Pruning phase consists at most $\sqrt{K}$ episodes, so resulting in at most $O(\sqrt{K})$ violation. 
* During the refinement phase, based on the result on page 16 of the supplemental material (Inequality (27)), we have with a high probability that the mixed policy used in the $t$th iteration has a regret bounded by $O(\frac{\log K}{(t-1)\sqrt{K}})$ for $t\geq 2.$ The same result holds for constraint violation, i.e., with a high probability, the constraint violation of the policy used in the $t$th iteration is bounded by  $O(\frac{\log K}{(t-1)\sqrt{K}}).$ The $t$th iteration includes $\sqrt{K}$ episodes and the refinement phase includes $\sqrt{K}$ iterations. Therefore, with a high probability, the total violation without canceling across episodes is $O(\sqrt{K}\log K).$
* The mixed policy used in the policy identification phase has a constraint violation bound of $O(\frac{\log K}{\sqrt{K}}).$ The policy is used for $K$ episodes, so the violation is bounded by   $O(\sqrt{K}{\log K}).$

Therefore, the total violation is bounded by $\tilde{O}(\sqrt{K})$ by using definition (5) in [Efroni et al., 2020]. We will add this in the revision.

Reviewer comment (Novelty):
The algorithm strongly relies on the Triple-Q algorithm, employing it as subroutine. Thus, the algorithmic novelty is partial.

Author response:
Our results hold if we replace Triple-Q with any model-free algorithm for CMDPs with sublinear regret and sublinear constraint violation. PRI can be viewed as a ``meta-algorithm'' that builds on any model-free CMDP algorithm with sublinear regret and constraint violation to achieve  $\tilde{O}(\sqrt{K})$ regret and constraint violation and to output a single stochastic policy.

Reviewer comment (Theory):
The assumption that the CMDP’s LP has a unique solution is strong. Since it is relaxed in the second part of the paper, I do not see any reason to focus half of the paper on this case. Same reasoning holds for assumption 3.

Author response:
We present the special case phase first for readability. Focusing on the special case where the problem has a unique optimal solution allows us to explain the key ideas behind the algorithm and makes the analysis easier to follow. We can directly present the general case as the reviewer suggested, but it will not save much space because the results we established for the single-solution case are used for the multi-solution case. Therefore, there is minimal repetition in the current structure.

Reviewer comment (Theory):
The theoretical results hold only for Large K, while no guarantees are provided if the number of episodes is small.

Author response:
The reviewer is correct that the result holds only for large $K.$ It is common that regret bounds in bandits and reinforcement learning hold only for large $K.$ Note that order-wise results such as $O(\sqrt{K})$ means it holds when $K$ is sufficiently. Big O notation defines the limiting behavior, i.e. when $K$ goes to infinity.

Reviewer comment (Theory):
In chapter 6, when the assumption on unique solution is relaxed, the authors introduce additional strong assumptions. For example, Assumption 4 states that the algorithm is given as input a lower bound on the probability of visiting every state-action pairs (when it is not 0) under the optimal policies belonging to the extreme point of the decision space.

Author response:
Assumption 4 is similar to Assumption 1. It states that if a state-action pair is visited under an optimal policy, then the visiting probability is lower bounded. We believe it is a reasonable assumption and easy to satisfy for problems with finite state and action spaces.

Reviewer comment (Theory):
Given that the main novelty of the paper concerns the model-free nature of the algorithm (indeed, model-based algorithms achieves better theoretical guarantees), the authors should devote more space clarifying which is the improvement in terms of the computational complexity of the algorithm proposed with respect to prior works.

Author response:
Thanks for the suggestion! Model-free algorithms require a memory complexity of $O(HSA)$ for maintaining the Q-Table  while model-based algorithms require a memory complexity of $O(HS^2A)$ for maintaining all transition probabilities. This is the main advantage of using model-free algorithms in practice. We will clarify this in the revision.

Reviewer comment (Theory):
Triple-Q assumes that salter condition holds. I assume this must be true even in your case, since Triple-Q is employed in PRI algorithm. Is it right?

Author response:
Yes, the reviewer is correct. We will clarify this in the revision.