Paper: Sinkhorn Distributional Reinforcement Learning (ICLR-2024)

Reviewer comment (Evaluation):
I am not quite convinced by the experimental results. Specifically, Figure 5 and the subsequent remark indicate that "sinkhornDRL achieves better performance across almost half of the considered game". However, it might be possible that both QRDQN and MMD each excel in roughly 50% of the games over the other. Consequently, sinkhorn should, on average, surpass either of the two in 50% of the games, as Sinkhorn essentially interpolates between these two algorithms. Observing Table 3 seems to draw the same conclusion: when Sinkhorn outperforms QRDQN, it often falls to MMD; conversely when it outperforms MMD, it tends to underperform relative to QRDQN. However, I admit that this is not absolute, as there are indeed some games where Sinkhorn surpasses both. But my concern is whether "better performance across half of the games" is enough.

Author response:
Let us summarize relevant results based on Table 3 to clarify your concerns. **Among 26 (22) games where SinkhornDRL outperforms QRDQN (MMDDRL), Sinkhron still performs better than MMDDRL (QRDQN) on 12 of them**.  Therefore, it is not sufficient to directly think that SinkhornDRL outperforms one and at the same time is inferior to the other. Generally speaking, although we can observe better performance on half of the games, the number of games on which a certain algorithm is superior to the others is only one of the important metrics to evaluate the capability of RL algorithms. We argue that the **Mean, Interquartile Mean(IQM), and median are shown in Table 2 and the learning curves in Appendix D are also very informative and more commonly used**. Table 2 showcases SinkrhornDRL is a generally better or competitive algorithm, and all learning curves in the Appendix substantiate that our algorithm is significantly better than the others in many games and competitive in the remaining ones.

Reviewer comment (Novelty):
There seem to be some related works that are missed. I'm not sure if all of them are relevant, but the author can check if they are related. Some remarks are attached to each of them.

Author response:
Thanks for pointing out these related works and we have cited them appropriately in the revised paper. Here we give our discussion.
	
* [1] is a Bayesian version of the distributional RL algorithm based on Wasserstein Variational Inference.
* [2] TV can dominate (as an upper bound) many metrics, including Wasserstein distance, however distributional Bellman operator is not contractive over TV [10]. This is why the theoretical results in the infinite horizon setting in [2] used average Wasserstein distance rather than TV.
* [3] is the first to study the advantages of distributional RL algorithms on the offline setting by imposing a conservative penalty term with the contraction guaranteed by Wasserstein distance.
* [4] is an extension paper of QR-DQN to prove the convergence of quantile TD in the TD learning setting instead of the dynamic programming setting by leveraging the non-smoothness analysis and new stochastic approximation theory. Akin to QR-DQN, in the first step our paper focuses on the contraction property in the dynamic programming setting and similarly we can also leave an extension paper to prove the TD learning under Sinkrhon divergence as future work. One thing we want to add here is that the TD learning under MMD and Sinkhorn is based on samples, whose convergence is supposed to be more easily guaranteed via the standard stochastic approximation theory than the quantile TD. We leave more rigorous analysis as the extension work in the future.

Reviewer comment (Novelty):
Li, Luchen, and A. Aldo Faisal. "Bayesian distributional policy gradients." — this paper proposes the policy gradient for distributional RL, and they use Wasserstein distance as well.

Author response:
Thanks for pointing out these related works and we have cited them appropriately in the revised paper. Here we give our discussion.
	
* [1] is a Bayesian version of the distributional RL algorithm based on Wasserstein Variational Inference.

Reviewer comment (Novelty):
Ma, Yecheng, Dinesh Jayaraman, and Osbert Bastani. "Conservative offline distributional reinforcement learning." — this paper learns conservative return distributions, which seems necessary in the offline setting. Their theoretical guarantees are also under the Wasserstein distance.

Author response:
Thanks for pointing out these related works and we have cited them appropriately in the revised paper. Here we give our discussion.
	
* [3] is the first to study the advantages of distributional RL algorithms on the offline setting by imposing a conservative penalty term with the contraction guaranteed by Wasserstein distance.

Reviewer comment (Theory):
I really appreciate that the author established some nice properties of sinkhorn divergence (e.g., theorem 1, proposition 1). However, it is still unclear to me why the authors proposed to use sinkhorn divergence. They claimed that it is an interpolation between Wasserstein distance and MMD, and thus, I am wondering why it is expected to be better. It would be great if the authors could provide a more either rigorous or intuitive explanation.

Author response:
The interpolation nature of Sinkhorn divergence helps it to **enjoy the advantages of both Wasserstein distance and MMD**, and thus the corresponding distributional RL algorithms are more likely to perform better in many environments. As mentioned in the second paragraph in the Introduction, compared with Wasserstein distance, 
* Sinkhorn divergence makes the computation of Wasserstein distance **tractable** by adding entropic regularization to overcome the curse of dimensionality issue, especially in the multi-dimensional reward function setting of RL, instead of using the quantile regression to approximate. It thus naturally avoids the non-crossing issue in quantile-based algorithms.
* Sinkhorn divergence uses the KL regularization to **encourage a "smoother" transport plan**, where the points are less tightly coupled to their specific targets in the optimal transport plan. This property is particularly beneficial in generating samples to approximate the current return distribution or in generative models, **ensuring that the generated samples cover the return distribution more uniformly**.

We provide [5-9] for further reference.

Reviewer comment (Writing):
The presentation needs to be polished since a lot of notations have not been introduced and may not be reading-friendly for people who are not familiar with distributional RL literature, as listed later.

Author response:
**A:** Thanks for this suggestion and we have fixed the presentation issues you pointed out in the revised version.

Reviewer comment (Reproducibility):
The introduction of the algorithms seems not sufficient, for instance, what is the next step after getting the Sinkhorn distance in Algorithm 1, such as gradient descent or other to minimize this distance?

Author response:
**A:** Thanks for pointing out this issue and **we additionally provide a full version of the Sinkhorn distributional RL algorithm in Algorithm 3 of Appendix I in the revised paper**. In particular, Algorithm 1 suggests the generic SinkhornDRL update, in which the Sinkhorn divergence needs to be evaluated via the differential Sinkhorn iterations in Algorithm 2 we introduced later.  After approximating the Sinkhorn divergence, we directly update the parameters $\theta$  in the network $Z_\theta$. Please refer to the full version of the Sinkhorn distributional RL algorithm in Algorithm 3 we just updated.

Reviewer comment (Theory):
Section 4.2 has a sequence of theoretical results, while more intuition will be helpful, such as the term $\overline{\Delta}(\gamma, \alpha)$ in Theorem 1(3) is represented in a very complicated way. So what do these terms mean and how large are they will be more helpful for the readers? So as the relationships to Gaussian or general kernels.

Author response:
**A:** Thanks for pointing out the interpretability issue in Theorem 1 (3). After some small modifications, we have provided a more interpretable version of Theorem 1(3) in the revised version. In particular, we further define a ratio $\lambda_{\mu, \nu}=\frac{\overline{\mathcal W} c, \varepsilon - \varepsilon \textbf{KL}(\Pi^*| \mu \otimes \nu)}{\overline{\mathcal W} c, \varepsilon}$, which **measures the proportion of optimal transport distances regardless of the KL regularization over the whole Sinkhorn distance** $\overline{\mathcal W} c, \varepsilon$. Finally, we obtain that the contraction factor has **a convex combination form between $\gamma^\alpha$ and 1 with the coefficient $\inf_{\mu, \nu} \lambda(\mu, \nu)$**, where $\overline{\Delta}(a, \alpha) = \gamma^\alpha \inf_{\mu, \nu}\lambda(\mu, \nu) + (1-\lambda(\mu, \nu))$ with $\lambda(\mu, \nu) \in (0, 1]$. The $\inf_{\mu, \nu} \lambda(\mu, \nu)$ characterizes the range of return distribution set in a given MDP, for which we prove that it is strictly greater than 0 and thus  $\overline{\Delta}(a, \alpha) <1$ strictly. **Intuitively, the contraction factor $\overline{\Delta}(a, \alpha)$ is MDP-dependent determined by the tight lower bound of the non-KL term ratio over the whole Sinkhorn divergence**, which is related to the range of the return distribution set.  Please refer to the updated paragraph immediately following Theorem 1 for a more detailed explanation. We have also proofread and updated the proof accordingly.
		
Moreover, for the relationships to differential kernels, we discussed them in the following paragraph "Consistency with Related Theoretical Contraction Conclusions" after Theorem 1. In particular, the contraction property is only guaranteed under the unrectified kernel $k_\alpha=-\Vert x - y \Vert^\alpha$, while for the Gaussian kernel, both Sinkrhon divergence and MMD can not guarantee such a contraction property. For instance, the MMDDRL paper points out a counterexample in their paper, demonstrating that MMD with a Gaussian kernel can not guarantee contraction in general.

Reviewer comment (Novelty):
As Sinkhorn is a well-known approach in optimal transport literature, it is curious what is the technical contribution of this paper, is it just an RL application inspired by Sinkhorn?

Author response:
**A:** The motivation of this paper is **two-fold**. Firstly, **from the perspective of distributional RL algorithms**, the choice of distributional divergence equipped with density estimation techniques has become the two key factors in designing new algorithms. It is a fact that the most commonly used distributional RL algorithms are based on MMD and Wasserstein distance. Since Sinkhorn has desirable properties, which interpolates the two distances, it is thus theoretically motivated to study the efficacy of Sinkhorn in a distributional RL context. 
	
On the other hand, **from the perspective of the optimal transport field**, since Sinkhorn divergence has been successfully applied to various machine learning models and contexts, including generative models, privacy, and adversarial robustness. It is also motivated to explore the application of Sinkhorn divergence in the RL setting, where we find it is more suitable in distributional RL.
	
Although our algorithm is well-motivated, there are still many challenges in designing Sinkhorn-based distributional RL algorithms both theoretically and empirically. Our papers address them comprehensively by proposing technically sound algorithms, studying the contraction property, and conducting extensive experiments for demonstration.

Reviewer comment (Writing):
Section 2.1 uses $\overset{D}{:=}$ without defining the notation of D.

Author response:
**A:** Thanks for pointing out these issues and we have fixed them in the revised version.

Reviewer comment (Writing):
$Z_{\theta^\star}$ has not been introduced.

Author response:
**A:** Thanks for pointing out these issues and we have fixed them in the revised version.

Reviewer comment (Theory):
There are concerns about the proof of Theorem 1. While the claims are believed to be correct, there are some parts that are less clear and should be clarified.

Author response:
**A:** We apologize for the confusion generated by some imprecise parts in the proof of Theorem 1 in the Appendix. You are right as the previous proof seems not to have involved a uniform convergence. Therefore, we have corrected the proof accordingly based on some optimal transport literature and notes. In particular, we replaced the proof in the language of $\epsilon, \delta$ in both part 1 and part 2 by providing a uniform convergence in the revised version. Please refer to the details in the paragraphs in red in Appendix B.
	
For part 3, we understand your concern as the minimizers can be the same although it is very unlikely to happen. To remedy this less rigorous proof, we devoted ourselves recently to a more rigorous proof and eventually, we figured out **this discrepancy is stemming from the finite return distribution set in a finite MDP**. This is related to the range of return distribution, coinciding with your comment. In particular, we define a ratio $\lambda(\mu, \nu)=\frac{\overline{\mathcal W} c, \varepsilon - \varepsilon \textbf{KL}(\Pi^*| \mu \otimes \nu)}{\overline{\mathcal W} c, \varepsilon}$, which measures the proportion of optimal transport distances **regardless of the KL regularization** over the whole Sinkhorn distance $\overline{\mathcal W} c, \varepsilon$. It turns out that $\inf_{\mu, \nu} \lambda(\mu, \nu) > 0$ strictly due to a finite set $\\{\lambda(\mu, \nu)\\}$ with a finite return distribution set in a given finite MDP. We believe this revised proof is more technically rigorous and the insight behind it is more intuitive. Let us know if you have any comments on this point.

Reviewer comment (Writing):
The paragraph labeled "Advantages over Quantile-based / Wasserstein Distance Distributional RL" is highly unpleasant to read. The "inline bulletpoint" style is not very easy to follow, and some concepts are not defined, which limits the utility of this paragraph for motivating SinkhornDRL.

Author response:
**A:** Thanks for pointing out this clarity issue. We re-organized this motivation part in the revised paper, by summarizing the advantages over Wasserstein and MMD into two points and one point respectively, and providing relevant references for some undefined concepts, including the non-crossing issue. We invite the reviewer to read and any further suggestion is also welcome.

Reviewer comment (Theory):
Regarding the contraction factor $\Delta(a,\alpha)$, there should be an explicit upper bound given (perhaps in terms of the range of returns). The fact that this term is strictly less than 1 relies on some discrepancy between Wasserstein and entropy-regularized Wasserstein, and it is not intuitive how large that discrepancy is. It is unclear whether we should expect $\Delta(\gamma,\alpha)$ to be larger or smaller than, say, $\gamma$ or $\sqrt{\gamma}$.

Author response:
**A:** This is a great suggestion. We devote ourselves to updating in the rebuttal period and end up with a more interpretable version of Theorem 1(3). In particular, we further define a ratio $\lambda(\mu, \nu)=\frac{\overline{\mathcal W} c, \varepsilon- \varepsilon \text{KL}(\Pi^* | \mu \otimes \nu)}{\overline{\mathcal W} c, \varepsilon}$, which measures the proportion of optimal transport distances **regardless of the KL regularization** over the whole Sinkhorn distance $\overline{\mathcal W} c, \varepsilon$. It turns out that $\inf_{\mu, \nu} \lambda(\mu, \nu) > 0$ due to a finite ratio set $\\{\lambda(\mu, \nu)\\}$ and a finite return distribution set in a finite MDP (yes, you are right, **the upper bound is explicitly related to the return distribution set**). This further implies that the contraction factor $\overline{\Delta}(a, \alpha) = \sup_{U, V} \overline{\Delta}^{U, V}(a, \alpha) \in [\gamma^\alpha, 1)$ (less than 1 strictly). Please refer to the updated paragraph immediately following Theorem 1 for a more detailed explanation. We have also proofread and updated the proof accordingly.

Reviewer comment (Evaluation):
Regarding the empirical results, it is unsettling that Table 2 reports only the statistics of the "Best" scores, and it is not actually clear what "Best" means.

Author response:
**A:** The best human normalized score is commonly used in (distributional) RL algorithms papers, including QR-DQN, MMD, and IQN. It refers to the highest (or peak) performance achieved by the RL algorithm during its training or evaluation process. The reason why we typically use best scores instead of average or median of scores is that **best scores help to demonstrate the peak capabilities of an RL algorithm**. It also shows what the algorithm can achieve under optimal conditions, which is particularly important in research and development contexts.

Reviewer comment (Evaluation):
The "ratio improvement" figures only show results for games where SinkhornDRL outperforms its competitors (with selection of games varying per competitor). The corresponding plots in the Appendix show that SinkhornDRL really only outperforms its competitors on roughly half (or slightly less than half) of the games.

Author response:
**A:** Since the superiority of typical distributional RL algorithms is more likely to vary across different games, it is thus very important to probe **in which environments our proposed algorithm tends to perform best**.  That is why we focus on the games where our algorithm particularly performs better than others in the ratio improvement figure. To avoid the bias of environment selection in evaluating algorithms, we also provide a comprehensive performance comparison across all games in the Appendix. Although SinkhornDRL performs better than MMD or QRDQN on almost half of the games, **the more valuable and comprehensive metrics, such as mean human normalized score and interquartile mean substantiate that our algorithm is generally better**. This is very intuitive as Sinkhorn divergence can find a sweet trade-off between QR-DQN and MMD, but it does not mean Sinkhorn divergence is overwhelmingly advantageous over QR-DQN or MMDDRL across (most of) all games.

Reviewer comment (Theory):
In the proof of Theorem 1, part 1 is not understood and the math is fairly imprecise. The definition of convergence leveraged between $\mathcal{W}^{c, \epsilon}$ and $\mathcal{W}^\alpha$ is sloppy — it is written as if these are scalar functions, but they are not. It is unclear if the convergence is uniform. Also, equation (14) does not establish the contraction any better than the claim that $\mathcal{W}^{c,\epsilon}\to\mathcal{W}_\alpha$ does, in the reviewer’s opinion. A contraction has not been shown. The same comments apply for the proof of part 2.

Author response:
**A:** We apologize for the confusion generated by some imprecise parts in the proof of Theorem 1 in the Appendix. You are right as the previous proof seems not to have involved a uniform convergence. Therefore, we have corrected the proof accordingly based on some optimal transport literature and notes. In particular, we replaced the proof in the language of $\epsilon, \delta$ in both part 1 and part 2 by providing a uniform convergence in the revised version. Please refer to the details in the paragraphs in red in Appendix B.

Reviewer comment (Theory):
The correctness of the proof of Theorem 1, part 3 relies on a hypothesis that the optimal coupling for $W_\alpha$ and $W^{c,\epsilon}$ cannot be the same (otherwise step (b) of equation (21) can be an equality). It is unclear if this hypothesis is known to be true, and if so, it should be cited.

Author response:
For part 3, we understand your concern as the minimizers can be the same although it is very unlikely to happen. To remedy this less rigorous proof, we devoted ourselves recently to a more rigorous proof and eventually, we figured out **this discrepancy is stemming from the finite return distribution set in a finite MDP**. This is related to the range of return distribution, coinciding with your comment. In particular, we define a ratio $\lambda(\mu, \nu)=\frac{\overline{\mathcal W} c, \varepsilon - \varepsilon \textbf{KL}(\Pi^*| \mu \otimes \nu)}{\overline{\mathcal W} c, \varepsilon}$, which measures the proportion of optimal transport distances **regardless of the KL regularization** over the whole Sinkhorn distance $\overline{\mathcal W} c, \varepsilon$. It turns out that $\inf_{\mu, \nu} \lambda(\mu, \nu) > 0$ strictly due to a finite set $\\{\lambda(\mu, \nu)\\}$ with a finite return distribution set in a given finite MDP. We believe this revised proof is more technically rigorous and the insight behind it is more intuitive. Let us know if you have any comments on this point.

Reviewer comment (Writing):
In the output of Algorithm 1, the second argument to $\overline{\mathcal{W}}^{c,\epsilon}$ should be $\{\mathcal{T}Z_j\}_{j=1}^N$.

Author response:
Thanks for pointing out these typos, and we have fixed them in the revised paper.

Reviewer comment (Writing):
Above equation (4), "supreme form of Sinkhorn divergence" should be "supremal form of Sinkhorn divergence".

Author response:
Thanks for pointing out these typos, and we have fixed them in the revised paper.

Reviewer comment (Writing):
It is unclear what the non-crossing issue is.

Author response:
**A:** In the quantile-based distributional RL, the learned quantile curves can not satisfy the non-decreasing property, especially at the early training stage, leading to **abnormal distribution estimates and reduced model interpretability**[1]. This issue can be largely mitigated by carefully adding the monotonicity constraint, e.g., adding an extra Non-crossing Quantile Logit Network[1]. By contrast, SinkhornDRL uses samples to approximate the return distribution, naturally avoiding this issue related to quantiles.

[1] Non-crossing quantile regression for deep reinforcement learning (Neurips 2020)

Reviewer comment (Theory):
It is unclear why the RKHS nature of MMD implies failure to capture geometry, since RKHS are Hilbert spaces.

Author response:
**A:** MMD computes the distance between two probability distributions as the difference in expectations of a function (belonging to the RKHS) evaluated at samples from these distributions. However, this nature causes the limitations of MMD to fully capture the data geometry:
* **The choice of kernel and resulting RHKS**. MMD needs to pre-specify the kernel, but the common kernels, such as Gaussian, might not adequately capture complex geometrical structures in the data.
* **Focus on Mean embedding**. MMD computes the difference in the mean embedding of two distributions in RKHS, however, only emphasizing the mean can overlook finer geometry properties of the distribution, including the higher-order moments, and shapes.
* **Linearity in RKHS**. Since RKHS is a linear space, relying on linearity is also a limitation as the geometric properties of distribution in the high-dimensional setting are more likely to be non-linear.

Reviewer comment (Novelty):
The main theoretical contribution can be seen as a simple combination of the architecture of MMD DRL and Sinkhorn divergence. The additional KL divergence term (mutual information) in the proposed loss is a key factor to improve the DRL algorithms except for two corner cases $\epsilon=0,\infty$. However, the role of this proposed element is not well explained in the main text.

Author response:
**A:** The mutual information in general renders Sinkhorn divergence to interpolate between Wasserstein distance and MMD, finding a sweet spot between their advantages. The mutual information leads to multiple advantages **over Wasserstein distance** summarized as follows. Here, we provide [1-6] for reference.
* Wasserstein distance suffers from the intractable computation and the curse of dimensionality. This regularization makes the optimization problem (of finding the optimal transport plan) strictly convex and thus **easier to solve**, and thus it significantly reduces computational complexity, allowing the evaluation in the **multivariate setting**, i.e., multi-dimensional reward setting in RL. Also, Sinkhorn divergence transforms the previous quantile-based Wasserstein distance algorithms into a sample-representation way, naturally avoiding the non-crossing issue in quantile-based algorithms and allowing more flexible statistics to represent the return distribution.
* It helps to find a **"smoother" transport plan** compared with the Wasserstein distance, where the points are less tightly coupled to their specific targets in the optimal transport plan or less sensitive to the specific arrangement of sample points.  This also implies that the Sinkhorn divergence can encourage more diverse mappings between distributions, which can be particularly beneficial in generative modeling or return distribution approximation in distributional RL, **ensuring that the generated samples cover the target distribution more uniformly**.
	
	
**For our conclusion made in the experiments**, firstly we keep the Wasserstein distance-based QR-DQN, MMDDQN, and SinkhornDRL in the same architecture, including the last output layer (200 quantiles, and 200 samples in MMD/Sinkhorn). As such, the only difference is the choice of distributional divergence equipped with different computation ways over the last layer output. Thus, the properties of distances will be crucial to explain the performance difference. Although we are aware of their different theoretical properties, it is still an open problem to explain which one performs better than others in each environment, which needs more knowledge about each Atari game. Although it is hard to establish the theoretical connection, we did extensive experiments, showing that SinkhornDRL tends to perform best in complex environments with a large action space and complicated environment. This statement is empirical, however, it provides some insight into the impact of different statistical distances with respective properties on real RL algorithms. **The mean and IQM metrics in Table 2 suggest our algorithm is a generally better algorithm than QR-DQN and MMD as Sinkrhorn divergence finds a sweet spot to trade-off their advantages.**

[1] Jean Feydy, Thibault S´ejourn´e, Franc¸ois-Xavier Vialard, Shun-ichi Amari, Alain Trouv´e, andGabriel Peyr´e. Interpolating between optimal transport and MMD using sinkhorn divergences.In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2681–2690.PMLR, 2019.
	
[2] Aude Genevay, Gabriel Peyr´e, and Marco Cuturi. Learning generative models with sinkhorn divergences.In International Conference on Artificial Intelligence and Statistics, pp. 1608–1617.PMLR, 2018
	
[3] Aaditya Ramdas, Nicol´as Garc´ıa Trillos, and Marco Cuturi. On Wasserstein two-sample testing and related families of nonparametric tests. Entropy, 19(2):47, 2017.
	
[4] Aude Genevay, L´enaic Chizat, Francis Bach, Marco Cuturi, and Gabriel Peyr´e. Sample complexity of sinkhorn divergences. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1574–1583. PMLR, 2019.
	
[5] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
	
[6] Richard Sinkhorn. Diagonal equivalence to matrices with prescribed row and column sums. The American Mathematical Monthly, 74(4):402–405, 1967.

Reviewer comment (Presentation):
It is hard to get an insight by introducing the Sinkhorn divergence. For example, the paper of MMD DRL provides the figure that explains the proposed scheme can estimate the high order moments better than the existing methods.

Author response:
**A:** As demonstrated in Proposition 1, with the Gaussian kernel, Sinkhorn divergence is equivalent to a **regularized moment matching scheme** that can still estimate the higher-order moments. Compared with MMD, the extra regularization term can be interpreted as a key factor in capturing the data geometry, which is the inherent advantage of Sinkhorn divergence over MMD.

Reviewer comment (Evaluation):
The main ablation study (sensitivity analysis) is provided without any detailed discussion. The authors state that the property of loss changes as the value of $\epsilon$ changes in the main theorem, but the empirical analysis and discussion are not provided well.

Author response:
**A:** We would like to clarify that we have provided a discussion about the sensitivity analysis regarding the varying $\epsilon$ in the sensitivity analysis part of **Section 5.2**:
		
In practice, a proper $\epsilon$ is preferable as an overly large or small $\epsilon$ will lead to numerical instability of Sinkhorn iterations in Algorithm 2, worsening its performance, as shown in Figure 3 (a). This implies that the potential interpolation nature of limiting behaviors between SinkhornDRL with QR-DQN and MMDDRL revealed in Theorem 1 may not be able to be rigorously verified in numerical experiments.
		
We have also conducted this sensitivity analysis on **Seaquest, Breakout, StarGunner, and Zaxxon games**, and have provided more detailed discussion in Appendix H.

Reviewer comment (Experiments):
The experiment protocol in this paper is not standard. Previous works almost conducted experiments in 200M iterations or the authors could have chosen the protocol of the paper, "DRL at the edge of statistical precipice" to validate the performance.

Author response:
**A:** As mentioned in the second paragraph of Section 5: Experiment, for a fair comparison, we build SinkhornDRL and all baselines based on a well-accepted PyTorch implementation (https://github.com/ShangtongZhang/DeepRL) of distributional RL algorithms, which has 3k stars. We run 40M frames instead of 200M **for computational convenience, but we report learning curves across all games and detailed scores for trustworthy results in the Appendix**. Although we agree that 200M frames would be stronger, we believe our careful and fair comparison of 40M frames with detailed learning curves and scores based on well-accepted implementation is sufficient for conveying convincing and trustworthy results.
		
In addition, we would like to clarify that we have followed the protocol of the paper DRL at the edge of the statistical precipice to validate the performance. The protocol includes three aspects: 
* **Interval estimation instead of point estimate**. We did so by providing all learning curves with the shading region indicating the standard deviation.
* **Performance profile/score distribution**. Although we have not provided an explicit score distribution, we have given all the scores across all considered distributional RL algorithms in Appendix F.
* **Interquartile mean (IQM) apart from mean and median**. We emphasized we have provided IQM results as well as mean, and median in Table 2, which is robust to outlier scores and more statistically efficient than Median.

Reviewer comment (Evaluation):
In Table 2 and Figure 2, the proposed method outperforms in terms of mean score but not in median score compared to baselines, and the proposed method shows remarkable performance in Venture and Sequest. This suggests that the proposed method is not a generally better algorithm but a specialist for Venture and Sequest. The authors should have stated why the proposed method is better in such environments.

Author response:
**A:** Firstly, the mean is more statistically efficient than the median, while the median is more robust to outliers. Thus, we have also provided results based on Interquartile Mean(IQM) to balance the efficiency and robustness in Table 2, and IQM results also demonstrate the superiority of our SinkhornDRL. 

Secondly, although Sinkhorn stands out in some games, e.g., Venture and Seaquest, **on raw scores, after the human normalization, the proportion of the resulting HMS for these two games is very limited across 55 games**. To provide more empirical details, below we rank all the human-normalized scores for each algorithm, and IQM(5%) removes the first two and last two games. You can check the result, which suggests **the largest scores in the remaining games are of a similar magnitude**. Thus, **our Sinkrhon algorithm is not a specialist for one or two games, but a generally competitive algorithm**.  For example, on the second-best game for SinkhornDRL, SinkhornDRL performs worse than the results on the other baselines' second-best game. (**Note that we average scores over games between Solaris and Demonattack (not included) for IQM(5%)**). However, by averaging all the (95%) games, SinkrhonDRL performs best on both mean (and IQM metrics) and is competitive on the median metric.

Reviewer comment (Experiments):
There exist more projection methods for distributional RL such as EDRL [A], but the provided baselines are not enough.

Author response:
**A:** EDRL is a variant of Quantile-regression DRL by changing the l1-based quantile regression to l2-based expectile regression, showing slight performance on Atari games (Figure 8 in EDRL[A]). Since Sinkrhon divergence interpolates Wasserstein distance, which is directly approximated by quantile regression instead of expectile regression, **QR-DQN, and MMD are the direct baselines to compare SinkhornDRL*. Moreover, as far as we know, as MMDRL is the state-of-the-art algorithm compared with variants of QR-DQN, including IQN and FQF, comparing MMDRL and QR-DQN with SinkhornDRL is sufficient. As also noted in the first paragraph, **proposing a new distributional RL family** and **increasing the model capacity within one family** are the main two research dimensions. Therefore, our research focuses on the first dimension, while directly comparing more theoretically related QR-DQN and MMDRL.

Reviewer comment (Theory):
It is hard to understand how reducing the mutual information helps to build a better projection operator. A detailed explanation is needed of why the mutual information between the return distribution and target distribution is important.

Author response:
**A:** By definition, Sinkhorn divergence is to find an optimal transport plan/coupling to minimize the transport costs between two distributions. Given two ground-truth return distributions, mutual information used in Sinkrhon divergence is to **encourage a "smoother" transport plan**, where the points are less tightly coupled to their specific targets in the optimal transport plan, compared with Wasserstein distance. This can be particularly beneficial in generating samples to approximate the current return distribution or generative modeling, ensuring that the generated samples cover the target return distribution more uniformly while minimizing the Sinkhorn divergence in terms of the parameterized return distribution $Z_\theta$. This also explains why Sinkrhon divergence potentially leads to a better projection operator compared with Wasserstein distance. Note that as we use deterministic samples in Sinkhorn divergence, **the updating of the sampling scheme (parameters in $Z_\theta$) in the current update step while minimizing the Sinkrhon divergence between two given return distributions, would directly affect the generated samples to approximate the return distributions in the next Bellman update**.