Paper: Interpreting Categorical Distributional Reinforcement Learning: An Implicit Risk-Sensitive Regularization Effect (ICLR-2024)

Reviewer comment (Theory):
how the epsilon in Eq. 3 is pre-specified? Should it not be dependent on the actual return distribution which cannot be known in advance? Am I correct in the understanding that you are actually not learning the full return distribution, but still its expectation albeit allowing some uncertainty around it?

Author response:
Eq.3 is to show that the return density can be decomposed via a pre-specified $\epsilon$, which controls the proportion of the expectation and the regularization part. Eq.3 does not imply that we are going to learn only based on the expectation, but to focus on the validity of the density decomposition. For example, in the sensitivity analysis in Figure 2, we are using the regularization term with different $\varepsilon$ to optimize the neural network in order to verify the impact of it.

Reviewer comment (Theory):
if so, it seems to me you're matching the expaction whilst expanding it according to a pre-defined level of uncertainty retained from the target distribution. In this case, how would you ensure that the so expanded distribution still has the same expectation/the correct expectation?

Author response:
Yes. The expectation can be inherently maintained by simply taking the expectation over both sides in Eq. 3. Since the first part maintains the same/correct expectation with the weight $1-\epsilon$, the remaining term accordingly has the same expectation with the weight $\epsilon$.

Reviewer comment (Theory):
if you are minimizing KL divergence, how would you ensure the return distribution model is and remains as a histogram which is count-based?

Author response:
The validity of the induced distribution is guaranteed in Proposition 1 for a certain range of $\epsilon$. The counted-based histogram is inherently maintained via the decomposition in Eq. 3.

Reviewer comment (Theory):
in Eq. 5, you're regularzing the cross-entropy between consecutive 'return distributions'. How is it connected to maximum-entropy RL which encourages 'policy' entropy or cross-entropy between the policy and a reference policy? There seems no bridge to me to be built between uncertainty in action selection and that in return estimate, the only connection perhaps being both can be approached by augmenting the reward function with the entropy term. In fact, I failed to understand the claim that your regularization is favouring target return distributions with large dispersion. It seems to me only to push the dispersion of the return distribution model to catch up with the dispersion of the target return distribution.

Author response:
Let us look at Eq. 10 to compare with MaxEnt RL in Eq. 7. Since we know $a_i = \pi(\cdot | s_i)$, optimizing the regularization term implicitly updates the policy $\pi$. By contrast, MaxEnt RL explicitly update the policy via maximizing the entropy in Eq. 7

In terms of Eq. 5, the regularization term in the value-based RL includes $q_\theta$, and optimizing Eq. 5 will directly affect the update of $q_\theta$. Note that $a_i =\arg\max_a \mathbb{E} [q_\theta^{s_i, a}]$, which therefore connects the return estimate with the action selection. 

We would like to clarify that we did not claim the regularization would certainly favor target return distribution with a large dispersion. **We state that it tends to do that as the original $q_\theta$ only captures the expectation and it tends to expand the distribution with a larger variance to match the distribution**. However, we agree your statement is safer and more accurate, which is that regularization in general helps to catch up with the dispersion of the target distribution. We have revised them appropriately in the updated paper.

Reviewer comment (Theory):
(minor) how and why a policy can be learned to reduce the intrinsic uncertainty of the environment, shouldn't it by definition be intrinsic and thus independent from the policy?

Author response:
The intrinsic uncertainty has been discussed in [1, 2], which is defined as the stochasticity of the environment, including the transition dynamics and stochastic reward function. Since distributional RL makes full use of the distribution information of return, it is deemed to reduce the intrinsic uncertainty of the environment for better performance.

Reviewer comment (Writing):
The paper is highly obscure and filled with ambiguous statements and typos that prevented me from validating most of its claims.

Author response:
**A:** We apologize for the confusion generated by the clarity issue. Thus, we have improved our claim following your suggestions throughout the paper as appropriate, and sincerely hope the current version is easier to follow. We will also be happy to edit the paper further based on any further helpful comments from the reviewers. The specific update corresponds to each clarity issue you raised as follows.

Reviewer comment (Theory):
The paper appears to support its theory using arguments that lack rigour or are purely contradictory.

Author response:
**A:** We apologize for the confusion generated by the clarity issue. Thus, we have improved our claim following your suggestions throughout the paper as appropriate, and sincerely hope the current version is easier to follow. We will also be happy to edit the paper further based on any further helpful comments from the reviewers. The specific update corresponds to each clarity issue you raised as follows.

Reviewer comment (Theory):
Example of lack of rigour — the use of the concept of "ideal case" mentioned at the bottom of page 2 to support the idea that Neural FQI converges to optimal Q-function, which is reused in the proof of proposition 3 to assume that neural FZI's iterate recover the Bellman update.

Author response:
**A:** Thanks for pointing out this issue. We have provided more specific explanations, respectively, instead of using ``the ideal case''. This is a commonly used assumption about the richness of function class in statistical learning theory. In Neural FQI, we explain as $\\{Q_\theta: \theta \in  \Theta\\}$ is sufficiently large such that it contains $\mathcal{T}^{opt} Q_{\theta^*}^{k}$, while in Neural FZI and proposition 3, we explain as $\\{Z_\theta: \theta \in  \Theta\\}$ is sufficiently large such that it contains $\{Y_i\}_{i=1}^n$. We also revised it accordingly to the updated version.

Reviewer comment (Theory):
Example of lack of rigour — a mismatch of assumption between the statement of proposition 3 and its proof, where the former assumes that the \( Y_i \)'s are representable while the latter assumes that \( E[Y_i] \) is.

Author response:
**A:** We would like to clarify that $\mathbb{E}\left[Y_i\right]$ can be representable is a corollary if $\\{Y_i\\}$ is representable as the $\mathbb{E}\left[Y_i\right]$ is (the limit of ) the  first component of $\{Y_i\}$'s pdf via our decomposition.

Reviewer comment (Theory):
Example of contradiction — Proposition 3 is contradicting itself in equation (6). The left equation claims that the minimizer over \( q_\theta \) converges to the k+1 iterate of the Q function in FQI while on the right claiming that it converges to the actual Q-value function.

Author response:
**A:** We apologize for this typo, where the action Q-value function should be replaced with the $k+1$ iterate of the Q function, which is thus consistent with the left equation. We have fixed the typo in the revised paper, where, specially, the correct form of the right equation  is updated as: 
	
$$\int_{-\infty}^{+\infty}\left|F_{q_\theta}(x)-F_{\delta_{ \mathcal{T}^{\text{opt}} Q^k_{\theta^*}(s, a)}}(x)\right| d x =o(\Delta)$$

Reviewer comment (Writing):
Typo/unclear statement — p2: defining the support of \( \hat{\eta} \) first as \( z_i \) then later by \( l_i \).

Author response:
$A:$ Thank you for noting these typos and we have fixed them appropriately. For example, we acknowledge that p.4 $\Delta_E^i$ represents the interval of $\mathbb{E}[Y_i]$ instead of the current return distribution and added a notation warning about the low case random variables. We also gave a more accurate definition of $\widehat{\mu}$.  
	
Please refer to the updated paper for more details. Please let us know if there are any other typos you had in mind for correction.

Reviewer comment (Writing):
Typo/unclear statement — Eq 1: \([y_i - Q_\theta(s_i,a_i)]^2\) instead of \([y_i - Q_\theta^k(s_i,a_i)]^2\).

Author response:
$A:$ Thank you for noting these typos and we have fixed them appropriately. For example, we acknowledge that p.4 $\Delta_E^i$ represents the interval of $\mathbb{E}[Y_i]$ instead of the current return distribution and added a notation warning about the low case random variables. We also gave a more accurate definition of $\widehat{\mu}$.  
	
Please refer to the updated paper for more details. Please let us know if there are any other typos you had in mind for correction.

Reviewer comment (Writing):
Typo/unclear statement — Eq. 2: similar issue but for \( Z_\theta^k(s_i,a_i) \).

Author response:
$A:$ Thank you for noting these typos and we have fixed them appropriately. For example, we acknowledge that p.4 $\Delta_E^i$ represents the interval of $\mathbb{E}[Y_i]$ instead of the current return distribution and added a notation warning about the low case random variables. We also gave a more accurate definition of $\widehat{\mu}$.  
	
Please refer to the updated paper for more details. Please let us know if there are any other typos you had in mind for correction.

Reviewer comment (Writing):
Typo/unclear statement — p.3 "As such, this ... shifting problems" needs rewritting.

Author response:
$A:$ Thank you for noting these typos and we have fixed them appropriately. For example, we acknowledge that p.4 $\Delta_E^i$ represents the interval of $\mathbb{E}[Y_i]$ instead of the current return distribution and added a notation warning about the low case random variables. We also gave a more accurate definition of $\widehat{\mu}$.  
	
Please refer to the updated paper for more details. Please let us know if there are any other typos you had in mind for correction.

Reviewer comment (Writing):
Typo/unclear statement — p.3 "the the".

Author response:
$A:$ Thank you for noting these typos and we have fixed them appropriately. For example, we acknowledge that p.4 $\Delta_E^i$ represents the interval of $\mathbb{E}[Y_i]$ instead of the current return distribution and added a notation warning about the low case random variables. We also gave a more accurate definition of $\widehat{\mu}$.  
	
Please refer to the updated paper for more details. Please let us know if there are any other typos you had in mind for correction.

Reviewer comment (Writing):
Typo/unclear statement — p.4 I suspect that in proposition 2 \( \Delta_E^i \) represents the interval that \( E[ R(s,a) + \gamma Z_k^\pi(s_i',\pi_Z(s_i')) ] \) and that generally there is a confusion between this expression and \( E[Z^\pi(s,a)] \).

Author response:
$A:$ Thank you for noting these typos and we have fixed them appropriately. For example, we acknowledge that p.4 $\Delta_E^i$ represents the interval of $\mathbb{E}[Y_i]$ instead of the current return distribution and added a notation warning about the low case random variables. We also gave a more accurate definition of $\widehat{\mu}$.  
	
Please refer to the updated paper for more details. Please let us know if there are any other typos you had in mind for correction.

Reviewer comment (Writing):
Typo/unclear statement — p.3 in an expression like \( Y_i = R(s_i,a_i)+\gamma Z_{\theta^*}^k(s_i',\pi_Z(s_i')) \) there needs to be a notation warning that lower case variables are given.

Author response:
$A:$ Thank you for noting these typos and we have fixed them appropriately. For example, we acknowledge that p.4 $\Delta_E^i$ represents the interval of $\mathbb{E}[Y_i]$ instead of the current return distribution and added a notation warning about the low case random variables. We also gave a more accurate definition of $\widehat{\mu}$.  
	
Please refer to the updated paper for more details. Please let us know if there are any other typos you had in mind for correction.

Reviewer comment (Writing):
Typo/unclear statement — p4 "for \( \forall \) k" should be "for all k".

Author response:
$A:$ Thank you for noting these typos and we have fixed them appropriately. For example, we acknowledge that p.4 $\Delta_E^i$ represents the interval of $\mathbb{E}[Y_i]$ instead of the current return distribution and added a notation warning about the low case random variables. We also gave a more accurate definition of $\widehat{\mu}$.  
	
Please refer to the updated paper for more details. Please let us know if there are any other typos you had in mind for correction.