Paper: Single-Trajectory Distributionally Robust Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
How do you justify the assumptions you made?

Author response:
We apologize for the confusing presentation of the results, particularly the organization of the assumptions.

For the main theoretical result (Theorem 3.3), which focuses on the asymptotic convergence of the DRRL algorithm, **we only require ONE assumption about the learning rate (Assumption C.10)**. This assumption can be easily satisfied by some common choices, such as $\zeta_1(n) = \frac{1}{1+n^{0.6}}$, $\zeta_2(n) = \frac{1}{1+n^{0.8}}$, and $\zeta_3(n) = \frac{1}{1+n}$.

The remaining assumptions are not directly related to the specific application of our proposed algorithm but serve to support our general three-timescale stochastic approximation framework. This framework is a side-product of our paper and aims to ensure its generality and independent usage value. **Regarding the soundness of the convergence results**, we verify that our proposed algorithm (Algorithm 1) meets all the necessary assumptions on Pages 23-24 in the proof of convergence. Consequently, its associated ODE has a unique asymptotically stable equilibrium.

These assumptions are standard and essential in the classical two-timescale stochastic approximation framework (refer to Section 6 in Borkar 2009), and we extend them to establish a three-timescale counterpart.

Reviewer comment (Writing):
What does the misspecified MDP in the paper mean?

Author response:
**Q.** Could you please clarify the meaning of the misspecified MDP in the paper?

**A.**  We apologize for not providing a formal definition of the misspecified MDP in the paper. In the context of distributionally robust reinforcement learning (DRRL), the main objective is to learn an optimal policy that is robust to unknown environmental changes. This becomes important when the transition model P and reward function r used during the training data collection differ from those in the test environment. In such cases, **we refer to the MDP environment (consisting of the transition model P and reward function r) used for training data collection as the misspecified MDP**. A detailed and rigorous definition can be found in Section 2.3. We appreciate your understanding and will ensure to provide clearer explanations in future work.

Reviewer comment (Novelty):
Badrinath & Kalathil (2021) and Roy et al. (2017) didn't use the R-contamination model. Their uncertainty set is a general one and covers yours. Both of them study the online setting and present asymptotic results. Why don't you compare with their works in experiments and compare their theoretical results with yours?

Author response:
**Q.** Comparison with Roy et al. (2017) and Badrinath & Kalathil (2021).

**A.** First, we would like to clarify that the uncertainty sets considered in Roy et al. (2017) and Badrinath & Kalathil (2021) **do not cover the ones in our work**, as they **do not require the transition model to be a valid probability transition** in the test environment. In fact, **their uncertainty set is equivalent to R-contamination set**. This leads to over-conservatism and significantly simplifies the theoretical difficulty and algorithmic design.

In their works, Roy et al. (2017) and Badrinath & Kalathil (2021) use the following ambiguity set:

$$
P_i^a := \\{x+ p_i^a \lvert x\in U_{i}^a \\},
$$

where $p_i^a$ is the unknown state transition probability vector from the state $i\in \mathcal{X}$ to every other state in $\mathcal{X}$. The confidence region $U_i^a$ must be constrained to ensure $x+p_{i}^a$ lies within the probability simplex. However, they drop the requirement and consider a proxy confidence region. 
Recall the definition of the R-contamination model,

$$
P_s^a=\\{(1-R) p_s^a+R q \mid q \in \Delta_{|S|}\\}, s \in S, a \in A, \text { for some } 0 \leq R \leq 1.
$$

Note that  $(1-R)p_s^a+R q$ may also lie outside the probability simplex. Thus under the relaxation, the ambiguity set used in Roy et al. (2017) and Badrinath & Kalathil (2021) are equivalent to the R-contamination model.
In contrast, our approach requires every element in the ambiguity set to lie within the probability simplex. 

This relaxation simplifies the algorithmic design and the convergence establishment. In particular, the $Q$-learning algorithm can be reformulated (see Equation (32) in Roy et al. (2017)) in terms of the operator $H$ as 
$$
Q_t(i,a) = (1-\gamma_t) Q_{t-1}(i,a) + \gamma_t (HQ_t(i,a) + \eta_t(i,a)),
$$
where $\eta_t(i,a)$ is some martingale noise in the non-robust $Q$-learning. 

In the robust Q-learning approaches of Roy et al. (2017) and Badrinath & Kalathil (2021), they also benefit from the relaxation mentioned earlier, as the proxy uncertainty set in Equation 10 in Roy et al. (2017) **does not depend on the incoming sample $j$**. Despite having sufficient samples, their algorithm may still penalize some cases that will never occur, as their ambiguity set lacks distribution-awareness. Thus they mainly focus on ellipsoid and parallelepiped as concrete examples.

In contrast, our approach maintains the worst-case distribution as a valid transition probability (referred to as distributionally aware) to prevent over-conservatism. As a result, $\eta_t(i,a)$ is not mean zero in our paper, as we require the transition to always fall within the probability complex. To address this challenge, our three-timescale algorithmic design, particularly the innermost loop and the second loop, along with the extended stochastic approximation framework, has been proposed.

Reviewer comment (Theory):
Why is the first equation on page 5 true?

Author response:
**Q.** Why the first equation on Page 5 is true?

**A.** Recall that the definition of $c_k(\rho)$ defined in Lemma 3.1 as  $c_k(\rho) = (1+k(k-1)\rho)^{1/k}\ge 1$. Define 

$$
f(\eta) = \eta-c_k(\rho)(\eta - \max_{a'}Q(s',a'))^+.
$$ 

Then the first order $f'(\eta) = 1 - c_k(\rho) \mathbb{1}\\{\eta \ge \max_{a'}Q(s',a')\\}$. 

We know $f'(\eta) = 1>0$ when $\eta <\max_{a'}Q(s',a')$ and $f'(\eta) = 0$ when $\eta \ge \max_{a'}Q(s',a')$. 

Thus $\max_{\eta\in \mathbb{R}}f(\eta) = f(\max_{a'}Q(s',a')) = \max_{a'}Q(s',a')$.

Plug it into Equation 5 we have 
$$
\begin{aligned}
r(s,a)+\gamma \sup_{\eta\in \mathbb{R}}\\{\eta-c_k(\rho)(\eta - \max_{a'}Q(s',a'))\\}
= r(s,a)+\gamma \{\max_{a'}Q(s',a')-c_k(\rho) \cdot 0\} = r(s,a) + \gamma \max_{a'}Q(s', a').
\end{aligned}
$$

Reviewer comment (Theory):
On page 6, why does keeping the learning speeds of $\eta$ and $Q$ different stabilize the training process?

Author response:
**Q.** The connection between different learning speeds of $\eta$ and $Q$ with the stability of the training process.

**A.**  Intuitively, $\eta(s,a)$ is independent of each state-action pair, whereas any state-action $Q$ value $Q(s,a)$ depends on other state-action pairs. Consequently, learning the optimal $Q$ value is more involved than learning the $\eta$ value. Due to this difference in their learnable nature, it is beneficial to set a relatively faster learning rate for $\eta$ to approximate the optimal value, while leaving the more challenging $Q$ function to be updated when $\eta$ is already accurate enough.

From a theoretical standpoint, as suggested by the stochastic approximation framework (e.g., Borkar 2009), one of the main analysis tools for multiple-timescale Q-learning, setting two different learning rates (with specific conditions) allows the training process to be tracked as an ODE system. This enables a full description of the system's dynamics and guarantees convergence. The difference in learning rates is a result of this theoretical understanding.

Reviewer comment (Theory):
The three-timescale regime used in algorithm 1 is complicated and provided with limited explanations. Why and how does it work?

Author response:
**Q.** Explain the three-timescale algorithm.

**A.**  The DRRL problem can be reduced to finding a fixed point for Equation (5), which involves finding a $Q$ value that satisfies the equation:

$$
Q(s, a)=r(s, a)+\gamma \sup _{\eta \in \mathbb{R}} \sigma_k(\max _{a^{\prime} \in \mathcal{A}} Q(\cdot, a^{\prime}),\eta).
$$

Since there are two variables, $\eta$ and $Q$, in this equation, we update them alternatively to approach the fixed point:

$$
\begin{aligned}
\eta_{n+1} &= \eta_n - \zeta_1 * \text{Gradient of } \eta_n,
\end{aligned}
$$
$$
Q_{n+1} =  r_n + \zeta_2 * \gamma \sigma_k(\max_{a'\in \mathcal{A}} Q(\cdot, a'), \eta_n).
$$

To obtain a better estimator for the gradient of $\eta_n$, we introduce two auxiliary variables, $Z_1$ and $Z_2$, to approximate the true ingredients in the gradient of $\eta_n$.

Combining all these components and arranging proper learning speeds for them, we have the following algorithm:

$$
Z_{n+1, 1} = \text{Update}(Z_{n,1})
$$

$$
Z_{n+1, 2} = \text{Update}(Z_{n,2})
$$

$$
\eta_{n+1} = \eta_n - \zeta_1 * \text{Gradient of } \eta_n
$$

$$
Q_{n+1} =  r_n + \zeta_2 * \gamma \sigma_k(\max_{a'\in \mathcal{A}} Q(\cdot, a'), \eta_n).
$$

The complexity in Algorithm 1 primarily arises from the explicit formula of the dual problem of the Cressie-Read divergence of the $f$ family and the gradient of $\eta_n$.

Reviewer comment (Theory):
In the proof, there are more than 10 assumptions and most of them don't have any explanation. Why do they hold in your case? Are they necessary? Are they reasonable in practical problems?

Author response:
**Q.** Too many assumptions.

**A.** We apologize for the confusing presentation of the results, particularly the organization of the assumptions.

For our main theoretical result (Theorem 3.3), which concerns the asymptotic convergence of the DRRL algorithm, **we only require ONE assumption about the learning rate (Assumption C.10)**. This assumption can easily be satisfied by some common choices, such as $\zeta_1(n) = \frac{1}{1+n^{0.6}}$, $\zeta_2(n) = \frac{1}{1+n^{0.8}}$, and $\zeta_3(n) = \frac{1}{1+n}$, and thus is almost as mild as the nonrobust $Q$ learning algorithm.

The remaining assumptions are not directly related to the specific application of our proposed algorithm but are included to support our general three-timescale stochastic approximation framework, which serves as a side-product in our paper. The purpose of including these assumptions is to ensure the framework is general enough to have independent usage value. These assumptions are standard and necessary in the classical two-timescale stochastic approximation framework (refer to Section 6 in Borkar 2009).

We confirm that our proposed algorithm (Algorithm 1) satisfies all the necessary assumptions on Pages 23-24 in the proof of convergence.

---

[1] Borkar, Vivek S. Stochastic approximation: a dynamical systems viewpoint. Vol. 48. Springer, 2009.

Reviewer comment (Novelty):
I guess the paper was written in parallel with the Panaganti 2022 (Robust offline) paper. However, as the Panaganti 2022 paper is published for over 6 months, it is better to compare and discuss the difference choices of uncertainty sets and problem settings. The authors claim that this paper is the first model-free DR RL paper in the literature, which is not true as the Panaganti 2022 is also model-free. To some extent, their paper considers a harder offline problem while this paper considers an online version.

Author response:
**Q.** Comparison with Panaganti et al. 2022.

**A.**  We appreciate your feedback and agree that our initial claim in the abstract may have been overstated.  We will make the necessary corrections to our claim.

We acknowledge the valuable contribution of the RFQI algorithm proposed by Panaganti et al. in 2022 and their novel work in addressing the DR offline RL problem with both theoretical guarantees and practical applications. Our primary focus is on the single trajectory scheme, which presents unique challenges compared to other papers, including Panaganti et al. 2022.

At the time of completion of our paper, the existing literature on model-free DRRL was limited, with the closest works being Panaganti et al. 2022 and Liu et al. 2022. We have already provided a thorough comparison of our algorithm with Liu et al. 2022 in the main text.

 We would like to kindly point out the primary distinctions between our proposed algorithm and Panaganti's approach:


1. Ambiguity Set: RFQI concentrates on the **total variation distance**, while we consider the Creese-Read family of $f$-divergence which can cover several commonly used divergence.
2. Technical Assumptions: RFQI heavily relies on the **fail-state** assumption, which assumes the existence of a state that yields a 0 reward regardless of the chosen action and must return to itself once entered. Such assumption aids to bypass the nonlinearity in the dual problem of the DRO problem with TV distance, i.e., the $\inf_{s''}V(s'')$ part in Equation 4 in Panaganti et al. 2022, and significantly simplify the estimation. This assumption may not be applicable in certain cases, such as infinite-horizon problems. Instead, the dual problem of the Creese-Read family of $f$-divergence is nonliear with respect the expectation part and we don't impose assumption to remove the nonlinearity. Instead we propose our multi-timescale algorithmic design to address it directly.
4. Data Collection Requirement: While RFQI aims to learn an optimal DR RL policy using a pre-collected offline dataset, our algorithm offers greater flexibility by allowing data to be fed in a single trajectory manner, which includes the pre-collected dataset setting.
5. Algorithmic Design: RFQI is based on the batch data scheme and develops their algorithm from value iteration with function approximation, alternating between updating the dual variable and the $Q$ value to ensure satisfactory optimization before updating the other variable. Conversely, our algorithm is grounded in the $Q$-learning approach and incrementally updates both the dual variable and the $Q$ value function simultaneously upon each data arrival, albeit with distinct learning rates.


   Our algorithm employs the Cressie-Read family of $f$-divergence and can accommodate several common divergences, including KL and $\chi^2$ divergence. The dual problem under this family is highly nonlinear with respect to the expectation, which is addressed by our multiple-timescale algorithmic design. Additionally, we do not impose extra assumptions for our chosen ambiguity set. 

|  | RFQI | DRQ (Ours) |
| --- | --- | --- |
| Ambiguity Set | Total Variation Distance | Cressie-Read family of f-divergence |
| Assumption for Ambiguity Set | Fail-State | No assumptions specific for ambiguity set |
| Data Collection | A batch of pre-collected data | Allows even single-trajectory data |
| Algorithmic Design | Value Iteration with alternative update between dual variable and Q | Q-learning with incremental and simultaneous update for dual variable and Q |

Reviewer comment (Novelty):
I don't think this is an issue when evaluating the contribution of this paper. However, it is better to state the contribution according to the latest literature (indeed, RFQI was mentioned in the experiment) and explicitly comment on the differences.

Author response:
**Q.** Comparison with Panaganti et al. 2022.

**A.**  We appreciate your feedback and agree that our initial claim in the abstract may have been overstated.  We will make the necessary corrections to our claim.

We acknowledge the valuable contribution of the RFQI algorithm proposed by Panaganti et al. in 2022 and their novel work in addressing the DR offline RL problem with both theoretical guarantees and practical applications. Our primary focus is on the single trajectory scheme, which presents unique challenges compared to other papers, including Panaganti et al. 2022.

At the time of completion of our paper, the existing literature on model-free DRRL was limited, with the closest works being Panaganti et al. 2022 and Liu et al. 2022. We have already provided a thorough comparison of our algorithm with Liu et al. 2022 in the main text.

 We would like to kindly point out the primary distinctions between our proposed algorithm and Panaganti's approach:


1. Ambiguity Set: RFQI concentrates on the **total variation distance**, while we consider the Creese-Read family of $f$-divergence which can cover several commonly used divergence.
2. Technical Assumptions: RFQI heavily relies on the **fail-state** assumption, which assumes the existence of a state that yields a 0 reward regardless of the chosen action and must return to itself once entered. Such assumption aids to bypass the nonlinearity in the dual problem of the DRO problem with TV distance, i.e., the $\inf_{s''}V(s'')$ part in Equation 4 in Panaganti et al. 2022, and significantly simplify the estimation. This assumption may not be applicable in certain cases, such as infinite-horizon problems. Instead, the dual problem of the Creese-Read family of $f$-divergence is nonliear with respect the expectation part and we don't impose assumption to remove the nonlinearity. Instead we propose our multi-timescale algorithmic design to address it directly.
4. Data Collection Requirement: While RFQI aims to learn an optimal DR RL policy using a pre-collected offline dataset, our algorithm offers greater flexibility by allowing data to be fed in a single trajectory manner, which includes the pre-collected dataset setting.
5. Algorithmic Design: RFQI is based on the batch data scheme and develops their algorithm from value iteration with function approximation, alternating between updating the dual variable and the $Q$ value to ensure satisfactory optimization before updating the other variable. Conversely, our algorithm is grounded in the $Q$-learning approach and incrementally updates both the dual variable and the $Q$ value function simultaneously upon each data arrival, albeit with distinct learning rates.


   Our algorithm employs the Cressie-Read family of $f$-divergence and can accommodate several common divergences, including KL and $\chi^2$ divergence. The dual problem under this family is highly nonlinear with respect to the expectation, which is addressed by our multiple-timescale algorithmic design. Additionally, we do not impose extra assumptions for our chosen ambiguity set. 

|  | RFQI | DRQ (Ours) |
| --- | --- | --- |
| Ambiguity Set | Total Variation Distance | Cressie-Read family of f-divergence |
| Assumption for Ambiguity Set | Fail-State | No assumptions specific for ambiguity set |
| Data Collection | A batch of pre-collected data | Allows even single-trajectory data |
| Algorithmic Design | Value Iteration with alternative update between dual variable and Q | Q-learning with incremental and simultaneous update for dual variable and Q |

Reviewer comment (Presentation):
The proposed algorithm 1 only works in the tabular setting. This is fine. The authors introduce a more practical algorithm using DQN type of learning scheme, which is provided in algorithm 4 in the Appendix. However, algorithm 4 looks almost the same as algorithm 1. Is this paper an older version? If yes, please provide the full algorithm 4 description in the future. More importantly, I believe algorithm 4 could potentially have a greater impact on DRRL problems; its connection and modifications with algorithm 1 could be further explained in details in the main paper.

Author response:
**Q.** Clarification about Algorithm 4.

**A.** We are sincerely sorry that mistakenly attaching the Algorithm 4 to the appendix, which is the same as Algorithm 1. **The more practical algorithm using DQN type of learning scheme is summarized into Algorithm 2**.

In particular, in Algorithm 2, we utilize the Deep Q-Network (DQN) architecture and neural networks as functional approximators for the dual variable and Q function in place of the tabular function in Algorithm 1. To improve training stability, we introduce target networks that are updated at a slower rate. We adopt a two-timescale update approach, which is employed to minimize the Bellman error for the Q network and maximize the DR Q value for the dual variable network. Even though this may introduce bias in the dual variable's convergence, resulting in a lower target value for the Q network, this approach serves as a robust update strategy for our DRRL problem, with further discussion in Appendix B.3. Moreover, Algorithm 1 updates on each data arrival, while the practical implementation uses a batch scheme, training the neural networks on a resampled subset of samples with a replay buffer retaining all previous samples.
We will move its connection and modifications with algorithm 1 to the main text.

Reviewer comment (Novelty):
Same to the weakness, please compare with Panaganti 2022 in details and explicitly.

Author response:
**Q.** Comparison with Panaganti et al. 2022.

**A.**  We appreciate your feedback and agree that our initial claim in the abstract may have been overstated.  We will make the necessary corrections to our claim.

We acknowledge the valuable contribution of the RFQI algorithm proposed by Panaganti et al. in 2022 and their novel work in addressing the DR offline RL problem with both theoretical guarantees and practical applications. Our primary focus is on the single trajectory scheme, which presents unique challenges compared to other papers, including Panaganti et al. 2022.

At the time of completion of our paper, the existing literature on model-free DRRL was limited, with the closest works being Panaganti et al. 2022 and Liu et al. 2022. We have already provided a thorough comparison of our algorithm with Liu et al. 2022 in the main text.

 We would like to kindly point out the primary distinctions between our proposed algorithm and Panaganti's approach:


1. Ambiguity Set: RFQI concentrates on the **total variation distance**, while we consider the Creese-Read family of $f$-divergence which can cover several commonly used divergence.
2. Technical Assumptions: RFQI heavily relies on the **fail-state** assumption, which assumes the existence of a state that yields a 0 reward regardless of the chosen action and must return to itself once entered. Such assumption aids to bypass the nonlinearity in the dual problem of the DRO problem with TV distance, i.e., the $\inf_{s''}V(s'')$ part in Equation 4 in Panaganti et al. 2022, and significantly simplify the estimation. This assumption may not be applicable in certain cases, such as infinite-horizon problems. Instead, the dual problem of the Creese-Read family of $f$-divergence is nonliear with respect the expectation part and we don't impose assumption to remove the nonlinearity. Instead we propose our multi-timescale algorithmic design to address it directly.
4. Data Collection Requirement: While RFQI aims to learn an optimal DR RL policy using a pre-collected offline dataset, our algorithm offers greater flexibility by allowing data to be fed in a single trajectory manner, which includes the pre-collected dataset setting.
5. Algorithmic Design: RFQI is based on the batch data scheme and develops their algorithm from value iteration with function approximation, alternating between updating the dual variable and the $Q$ value to ensure satisfactory optimization before updating the other variable. Conversely, our algorithm is grounded in the $Q$-learning approach and incrementally updates both the dual variable and the $Q$ value function simultaneously upon each data arrival, albeit with distinct learning rates.


   Our algorithm employs the Cressie-Read family of $f$-divergence and can accommodate several common divergences, including KL and $\chi^2$ divergence. The dual problem under this family is highly nonlinear with respect to the expectation, which is addressed by our multiple-timescale algorithmic design. Additionally, we do not impose extra assumptions for our chosen ambiguity set. 

|  | RFQI | DRQ (Ours) |
| --- | --- | --- |
| Ambiguity Set | Total Variation Distance | Cressie-Read family of f-divergence |
| Assumption for Ambiguity Set | Fail-State | No assumptions specific for ambiguity set |
| Data Collection | A batch of pre-collected data | Allows even single-trajectory data |
| Algorithmic Design | Value Iteration with alternative update between dual variable and Q | Q-learning with incremental and simultaneous update for dual variable and Q |

Reviewer comment (Evaluation):
For the experiments in Figure 5. It seems the proposed algorithm DDQR has a very similar performance compared to the existing robust algorithm SR-DQN. It will be helpful to add more discussion about this.

Author response:
**Q.** Disscussion about the similar performance of the proposed DDQR algorithm with the existing robust algorithm SR-DQN.

**A.** The SR-DQN algorithm demonstrates comparable performance to our proposed DDQR method, primarily when environmental parameters are perturbed (e.g., FMP perturbation in the Cartpole environment and EPP perturbation in the LunarLander environment). In these perturbations, even if the force magnitude (FMP) or engine power (EPP) is altered in the test environment, the transition remains almost deterministic, meaning only one state transitions when an action is taken under the current state. Consequently, these perturbations maintain a similar level of randomness as the original environment, which might not substantially degrade the learned policy. As a result, the soft-robustness principle employed in SR-DQN may be sufficient to manage the perturbation, leading to similar performance as our proposed algorithm.

However, when it comes to action perturbations in both the Cartpole and LunarLander environments, multiple states can be transitioned as different actions are deployed in the environment due to the perturbation. This increases the randomness level, and SR-DQN is notably outperformed by our DDQR algorithm, as it does not provide adequate robustness.

It is worth mentioning that SR-DQN does perform well in certain situations, as demonstrated in the Cartpole experiment by Panaganti et al. (2022).

---

[1] Panaganti, Kishan, et al. "Robust reinforcement learning using offline data." Advances in neural information processing systems 35 (2022): 32211-32224.

Reviewer comment (Experiments):
As mentioned in the algorithm, the three-timescale serves as a key role in the algorithm to ensure convergence. So it will be better to introduce what is the three learning rates that the practical algorithm uses. And ablation study using different learning rates will give a message about whether the algorithm is sensitive to the learning rate.

Author response:
**Q.** More details about the three learning rates in the practical algorithm and conduct an ablation study to assess its sensitivity to learning rates.

**A.** Our practical algorithm is updated in a two-timescale as the neural network already introduce a nonnegligible bias in the estimation, we allow the update of the dual variable from a biased gradient estimator and thus drop the usage of another timescale. We choose $2.5e^{-4}$ as the learning rate of the $Q$ network and $2.5e^{-3}$ as the learning rate of the dual variable and use the Adam optimizer to search for the optimal values. 

We conduct an additional ablation study to assess the robustness of our DDRQ. Utilizing the same experimental setups employed in the CartPole and LunarLander environments, as described in Section 4.2, we vary the learning rates for both the $Q$ network and the dual variable network. The summarized experimental results are presented in the following tables. For the CartPole environment, we test five groups of learning rates: $\{(10^{-4}, 10^{-3}), (2.5\times 10^{-4}, 2.5\times 10^{-3}), (5\times 10^{-4}, 5\times 10^{-3}), (10^{-4}, 10^{-2}), (10^{-3}, 10^{-2})\}$. Among these, the pair $(2.5\times 10^{-4}, 2.5\times 10^{-3})$ was previously presented in Section 4.2, and we rescale it to $(10^{-4}, 10^{-3})$, $(5\times 10^{-4}, 5\times 10^{-3})$, and $(10^{-3}, 10^{-2})$. While maintaining the same learning rates for the $Q$ network and the dual network, we vary their relative ratio by examining the pair $(10^{-4}, 10^{-2})$. Our experimental results suggest that our DDRQ exhibits relatively stable performance under different choices of learning rates, even when their ratio is altered. However, a larger learning rate in $Q$ network may render poorer performance, e.g., $(10^{-4}, 10^{-2})$ and $(10^{-3}, 10^{-2})$ pairs in Cartpole Action Perturbation experiment (Table 2). Also altering the ratio of the learning speeds may affect the performance, see $(10^{-4}, 10^{-2})$ in Cartpole FMP perturbation experiment (Table 1). 
It is important to note that for the more complex LunarLander environment, the pair $(10^{-4}, 10^{-3})$ is too small for our algorithm to converge to an optimal policy, while $(10^{-3}, 10^{-2})$ is too large for convergence within our two-timescale approximation scheme. Consequently, we only report the remaining pairs in Tables 3 and 4.

  | % Change | Learning Rates | $(10^{-4}, 10^{-3})$ | $(2.5\times 10^{-4}, 2.5\times 10^{-3})$ | $(5\times 10^{-4}, 5\times 10^{-3})$ | $(10^{-4}, 10^{-2})$ | $(10^{-3}, 10^{-2})$ |
|---|---|---|---|---|---|---|
| -80 |  | 196.8 (± 2.0) | 196.6 (± 4.9) | 195.9(± 2.4) | 168.9 (± 70.1) | 197.5 (± 5.13) |
| -70 |  | 197.7 (± 2.1) | 198.5 (± 2.1) | 198.3(± 5.8) | 171.8 (± 44.6) | 196.7 (± 2.13) |
| -60 |  | 200.0 (± 0.0) | 198.8 (± 1.9) | 200.0(± 0.0) | 171.2 (± 84.2) | 200.0 (± 0.0) |
| -50 |  | 200.0 (± 0.0) | 200.0 (± 0.0) | 200.0(± 0.0) | 178.0 (± 46.1) | 200.0 (± 0.0) |
| -40 |  | 200.0 (± 0.0) | 199.6 (± 0.9) | 200.0(± 0.0) | 178.8 (± 52.2) | 200.0 (± 0.0) |
| -30 |  | 200.0 (± 0.0) | 200.0 (± 0.0) | 200.0(± 0.0) | 184.7 (± 31.2) | 200.0 (± 0.0) |
| -20 |  | 199.8 (± 0.8) | 200.0 (± 0.0) | 200.0(± 0.0) | 189.1 (± 22.1) | 200.0 (± 0.0) |

Table 1: **Cartpole FMP Perturbation with Varied Learning Rates:** The first element in the learning rate tuple pertains to the Q network, while the second corresponds to the dual variable network. Each experiment is replicated 100 times, and the mean and standard deviation of the returns are reported.


| % Random | Learning Rates | $(10^{-4}, 10^{-3})$ | $(2.5\times 10^{-4}, 2.5\times 10^{-3})$ | $(5\times 10^{-4}, 5\times 10^{-3})$ | $(10^{-4}, 10^{-2})$ | $(10^{-3}, 10^{-2})$ |
|---|---|---|---|---|---|---|
| 0 |  | 200.0 (± 0.0) | 200.0 (± 0.0) | 200.0(± 0.0) | 196.6 (± 3.7) | 200.0 (± 0.0) |
| 10 |  | 200.0 (± 0.0) | 200.0 (± 0.0) | 200.0(± 0.0) | 157.5 (± 35.3) | 200.0 (± 0.0) |
| 20 |  | 194.9 (± 3.8) | 200.0 (± 0.0) | 196.4(± 5.9) | 166.7 (± 26.6) | 192.1 (± 5.8) |
| 30 |  | 187.3 (± 19.1) | 199.3 (± 1.8) | 197.5(± 2.4) | 149.4 (± 43.7) | 192.8 (± 6.7) |
| 40 |  | 164.4 (± 25.1) | 180.7 (± 17.8) | 180.5(± 21.38) | 124.7 (± 39.3) | 151.7 (± 35.3) |
| 50 |  | 121.7 (± 46.9) | 131.8 (± 53.8) | 133.1(± 41.4) | 72.3 (± 44.2) | 124.0 (± 73.4) |
| 60 |  | 88.3 (± 73.8) | 84.7 (± 61.2) | 81.4(± 68.7) | 43.3 (± 43.7) | 67.3 (± 41.9) |
| 70 |  | 65.3 (± 54.9) | 66.2 (± 65.1) | 59.4(± 57.4) | 37.5 (± 48.2) | 57.7 (± 49.8) |

Table 2: **Cartpole Action Perturbation with Varied Learning Rates:** The first element in the learning rate tuple pertains to the Q network, while the second corresponds to the dual variable network. Each experiment is replicated 100 times, and the mean and standard deviation of the returns are reported.

Reviewer comment (Theory):
As the update of $\eta$ is independent for each $(s,a)$ pair, will the proposed algorithm works for $s$-rectangular cases?

Author response:
**Q.** The potential of the proposed algorithm to work in s-rectangular setting.

**A.** We agree with the reviewer's insight that our proposed algorithm can work in s-rectangular setting by tracking the optimal dual variable for each state, rather then for each state-action pair in the current sa-rectangular setting.

Reviewer comment (Reproducibility):
In Section 4.2, a practical implementation of DQR is utilized for the experiments. How is this implemented? It would be nice to discuss such implementation in the main text.

Author response:
**Q: Discussion about the practical implementation of DQR algorithm.**

**A:** We have deferred the discussion of the specific implementation to Appendix B.3 and summarized into Algorithm 2. In this answer, we restate the implementation details, particularly the differences from the DRQ algorithm (Algorithm 1), and will move the main part of the discussion to the main text.

Instead of the tabular function in Algorithm 1, we adopt the Deep Q-Network (DQN) architecture (Mnih et al., 2015) and choose neural networks as functional approximators for the dual variable and the Q function. To enhance training stability, we introduce another set of neural networks as the corresponding target networks, which are updated at a slower rate. Due to the existence of approximation error, we adopt a two-timescale update approach: our Q network aims to minimize the Bellman error, while the dual variable $\eta$ network strives to maximize the DR Q value. The two-timescale update approach could introduce bias in the convergence of the dual variable, and thus the dual variable η may not be the optimal dual variable for the primal problem. Given the primal-dual structure of this DR problem, this could render an even lower target value for the Q network to learn. This approach can be understood as a robust update strategy for our original DRRL problem. More discussion about the validity of this update strategy can be found in Appendix B.3.

Another significant difference is in the update scheme. While in Algorithm 1, we update on each data arrival, in the practical implementation, the neural networks are updated in a batch scheme, i.e., trained on a resampled subset of samples with a replay buffer keeping all the previous samples.