Paper: CIM: Constrained Intrinsic Motivation for Reinforcement Learning (ICLR-2024)

Reviewer comment (Writing):
In the introduction, maybe focus early on the latent skill, why is it important, and what is the bias and what are its implications, e.g., why is it bad?

Author response:
A1: Thanks for the valuable suggestion. Following the reviewer's suggestion, we have discussed, in the Introduction section, the latent skill in competence-based IM methods in RFPT tasks and the bias introduced by knowledge- and data-based intrinsic rewards in EIM tasks in the revision. Especially, a policy conditioned on latent skill variables can consistently and meaningfully alter the environment's states, such as walking, jumping, flipping, or pushing. This makes it well-suited for efficient fine-tuning to solve downstream tasks. As for the bias introduced by intrinsic objectives in EIM tasks, intrinsic objectives can lead to superfluous exploration even when the task rewards are already accessible. This distraction, induced by intrinsic objectives, can deteriorate the performance of the agent. Consequently, it can impede the wider application of knowledge- and data-based IM methods.

Reviewer comment (Novelty):
The work primarily focuses on reward shaping which can be considered a heuristic development (e.g., eqn 7) upon the prior works. However, the authors also tried to give intuitions defending their choices. I am unable to judge contribution/novelty here.

Author response:
A2: Please allow us to clarify that although the total reward $r_e + \tau_k r_i$ in EIM tasks can be viewed as reward shaping, our primary focus in RFPT tasks is on pre-training an agent without any task reward $r_e$ available, which does not fall into the domain of reward shaping since there is no task reward to shape. Actually, our CIM for RFPT is more related to unsupervised skill discovery. What's more, we design a novel alignment loss function (Eqn. (7)) based on CPC theory to maximize the MI between the state representation and the latent skill variable. We provide a thorough comparison with previous choices (Eqn. (6)) for the alignment loss function to highlight the distinction between our approach and prior works. We also conducted an ablation study on the choice of the alignment loss function to demonstrate its impact on the performance of CIM. As shown in Table 3, CIM for RFPT with our novel alignment loss function $l_i^{\text{CIM}}$ exhibits superior performance in Ant. Apart from maximizing the lower bound of $I(\rm{s};\rm{z})$ by minimizing the alignment loss defined in Eqn. (7), we also propose maximizing the lower bound of the conditional entropy $H(\rm{s}|\rm{z})$ via Eqn. (10). By alternately maximizing the lower bounds of both $I(\rm{s};\rm{z})$ and $H(\rm{s}|\rm{z})$, CIM for RFPT significantly improves the state coverage and promotes the discovery of dynamic and diverse skills compared with all 15 baseline methods, as shown in Fig. 1, Fig. 2, and Table 1. Besides state coverage and skill diversity, our CIM can also achieve a 90x faster pre-training speed than LSD and the highest fine-tuning score in the Walker domain of URLB.

Reviewer comment (Theory):
The presentation of theoretical results is not so sharp. Need to say about s,z,g. Is it true for any s,z,g? Are there some assumptions on g and what does "with equality" mean?

Author response:
A3: Thanks for the valuable comment. Lemma 1 and Theorem 2 is true for any $\rm{s},\rm{z}$, and any deterministic function $g(\rm{s})$. Please allow us to provide the detailed proof of Lemma 1 as follows to show why $g$ is assumed to be deterministic. As suggested in the latest ICLR latex template, we use $\rm{s}$ and $\rm{z}$ for variables and use $z$ for a concrete sample. 

*Proof of Lemma 1.*  Observing that $H(\rm{s},g(\rm{s}))|\rm{z}=$$z) = H(\rm{s}|\rm{z}=$$z) + H(g(\rm{s})|\rm{s},\rm{z}=$$z) = H(g(\rm{s})|\rm{z}=$$z) +  H(\rm{s}|g(\rm{s}),\rm{z}=$$z)$, and $H(g(\rm{s})|\rm{s},\rm{z}=$$z)=0$ ($g$ is deterministic), we have $H(\rm{s}|\rm{z}=$$z) - H(g(\rm{s})|\rm{z}=$$z) = H(\rm{s}|g(\rm{s}),\rm{z}=$$z) \ge 0$. If and only if $g$ is invertible ($H(\rm{s}|g(\rm{s}),\rm{z}=$$z) = 0$), we have $H(\rm{s}|\rm{z}=$$z) = H(g(\rm{s})|\rm{z}=$$z)$.

Thus, the condition "with equality" means that $H(\rm{s}|\rm{z}=$$z) = H(g(\rm{s})|\rm{z}=$$z)$ if and only if we can deterministically infer $s$ given $g(s)$, i.e., $g$ is invertible, or in other words, $g$ is injective on the support of the state distribution $d_\pi(s)$. We have added the proof of Lemma 1 in Appendix F.2.

Reviewer comment (Theory):
On page 5, how are the last 4 lines related to eqn 10? I got a bit lost there.

Author response:
A4: The last 4 lines are aimed at explaining how the intrinsic reward function can be derived from the convex intrinsic objective, i.e. the proof sketch for $r_i \propto \nabla_{d_\pi} L(d_\pi)$. The detailed proof is provided by Hazan via the Frank-Walfe algorithm. Based on this, we can directly derive the intrinsic reward of CIM $r_i^{\text{CIM}}$ for RFPT after designing the convex intrinsic objective of CIM $H(g(\rm{s})|z)$, that is, $r_i^{\text{CIM}} \propto \nabla_{d_\pi} H(g(\rm{s})|z) = \nabla_{d_\pi} (\sum_z\sum_s -d_\pi \log d_\pi) = -\log d_\pi - 1$. By applying the non-parametric $\rm{k}$-NN estimator, we obtain $r_i^{\text{CIM}} \propto - \log{1/\|g(s) - g(s)^\rm{k}\|}$, where $g(s)^\rm{k}$ stands for the $\rm{k}$-th nearest neighbour of $g(s)$. To ensure stability during the training process, we utilize the average version of $r_i^{\text{CIM}}$, as defined in Eqn. (10). This average version has been shown to be more stable for intrinsic reward estimation by previous works like APT (Liu & Abbeel 2021) and MOSS (Zhao et al. 2022).

Reviewer comment (Theory):
What are typical latent skills, and how is policy conditioned on it? Is it already a priori known which skills we want to learn?

Author response:
A5: As answered in Q1, latent skills can be understood as the underlying abilities or action sequences that an agent can perform to consistently and meaningfully interact with the environment. For instance, as shown in Fig 1, the latent-conditioned policy learned via our CIM can control the Ant to run in any direction in the plane, which can be easily used for zero-shot goal-reaching tasks or as low-level skills for complex navigation tasks like AntUMaze. As stated in Section 3.1.2, following Eqn. (8), typically, the latent skill can be sampled from a prior distribution $p_z$. When there is no prior for the skills, a uniform distribution $\mathcal{U}^{n_z}(-1,1)$ is a common choice for sampling continuous skills, where $n_z$ is the skill dimension.

To make the policy condition on the latent skill, we concatenate the current state and the sampled skill to form an input vector $[s,z]$ for the policy network $\pi(a|s,z)$.

In this work, we do not rely on any prior knowledge or feature engineering for unsupervised skill discovery to make our algorithm more general.

Reviewer comment (Writing):
What is N in MI lower bound (above sec 3.1.2)?

Author response:
A7: The $N$ stems from the Contrastive Predictive Coding (CPC) theory. Specifically, $N$ is the total number of samples for estimating the MI. Especially, this approximation becomes more accurate as $N$ increases; that is, the more negative samples, the more accurate the MI lower bound. We have added the definition of $N$ in the revision. Empirically, we found that $N=128$ is sufficient for our experiments.

Reviewer comment (Writing):
Typos: Page 5 Walfe -> Wolfe, sec 4.3 dynamic is -> dynamics is, Lemma 1 and Theorem 2 extra bracket ).

Author response:
A8: Thanks for the valuable suggestions. We will fix the typos in the revision.

Reviewer comment (Novelty):
Comment: The paper studies an interesting problem of maximizing mutual information. It is related to experiment design kind of objectives or more widely maximizing submodular functions under MDP constraints. These functions capture that visiting the same state will result in reduced rewards and hence naturally encourage exploration to gain higher rewards. The submodular functions can thus encode state coverage, entropy maximization, and Mutual information (D-design) kind of objectives. There are works on submodular reinforcement learning that might be interesting and related to the authors. The work is also related to convex RL, but I see two papers cited (Hazan & Mutti) in that direction, so I believe the authors are aware of it.

Author response:
A9: Thanks for your comments that our work is related to more broad optimization problems, including submodular RL and convex RL. This is the first time we heard of submodular RL, and we are happy to explore this promising domain in our future work. As you mentioned, we are aware of convex RL from Hazan and Mutti, and our work was originally inspired by Hazan, a data-based IM method that maximizes state entropy in the framework of convex RL.

Reviewer comment (Experiments):
Experiments performed for EIM tasks are a bit insufficient. Only on two environments, one directly trains using the proposed intrinsic reward while one trains a meta-controller on the top of learned skills. It would be more convincing to test on more environments.

Author response:
A1: We chose two representative environments to illustrate the effectiveness of CIM for EIM in the main body of the paper due to space constraints. Here, we conduct more experiments to demonstrate the performance of CIM for EIM across four sparse-reward locomotion tasks (designed by Mutti et al., 2021): SparseHalfCheetah (SHC), SparseAnt (SA), SparseHumanoidStandup (SHS) and SpraseGridWorld (SGW). The test-time average episode rewards $J_e(d_\pi)$ are reported in the table below:

| **Scheduler** | **SHC** | **SA** | **SHS** | **SGW** |
|---|---|---|---|---|
|$\tau_k^{\text{C}}$ | 0.27 | 0.74 | 0.18 | 0.01 |
|$\tau_k^{\text{CIM}}$ | **1** | **0.97** | **1** | **1** |

As shown in the table, $\tau_k^{\text{CIM}}$ effectively reduces the bias introduced by intrinsic rewards, thereby enhancing performance in EIM tasks. We also draw the curves of $\tau_k^{\text{CIM}}$ in all EIM tasks to demonstrate that $\tau_k^{\text{CIM}}$ gradually tends to zero in Appendix F.2.

Reviewer comment (Evaluation):
In section 4.2, State Coverage paragraph, APT and CIC outperform CIM on two tasks, FetchPush and FetchSlide. Do you have any intuition as to why these two methods are particularly better in these two tasks?

Author response:
A2: FetchPush and FetchSlide are position-as-input manipulation tasks that feature less complex environmental dynamics compared to the torque-as-input locomotion tasks Ant and Humanoid. As a result, both the data-based IM method APT (Liu & Abbeel 2021) and the competence-based methods CIC (Laskin et al. 2022), CSD (Park et al. 2023), and CIM can maximize the state coverage with only 4M samples (5x less than Ant). However, the skills learned via our CIM are more interpretable than those acquired through APT and CIC. As depicted in Fig. 2, the trajectory's direction on the $x−y$ plane consistently changes in line with the skill direction (observed as a smoother color transition). This interpretation can also be seen in Fig 1. We have added "position-as-input" and "torque-as-input" in the revision to imply the complexity of dynamics in these tasks.

Reviewer comment (Presentation):
In Fig.3(a), in which task these discrete skills are learned? Also Ant or some other tasks?

Author response:
A3: In Fig. 3(a), we present a visualization of the discrete skills learned using CIM for RFPT in the Ant task. The visualization of the discrete skills learned in Humanoid is similar.

Reviewer comment (Presentation):
S_T, s’, z from table 1 are not explained in the caption, although they are here and there in the paper, I do think it would be clearer to add them in the caption as well.

Author response:
A4: Thanks for your valuable suggestions. We will revise our paper accordingly. We apologize for any confusion caused by the omission of the definitions for $s_T$, $s'$, and $z$ in Table 1 due to page limits. Specifically, $s_T$ represents the last state in one trajectory, where $T$ denotes the final time step in one episode. $s'\sim P(s'|s,a)$ is the subsequent state transitioned from the current state $s$ when action $a$ is taken. Lastly, $z$ is the latent skill. Following your suggestion, we have included these definitions in the caption of Table 1 to improve clarity.

Reviewer comment (Writing):
The paper is hard to follow because there are too many abbreviations of previous approaches, requiring the reader to go back and forth to avoid missing important information. The authors should replace these abbreviations with in-text citations if possible.

Author response:
Thanks for your helpful suggestion. We will revise our paper accordingly. To thoroughly summarize and compare previous Intrinsic Motivation (IM) methods, we had to include all 15 methods, which resulted in numerous abbreviations. As the reviewer suggested, we have replaced them with in-text citations in the revision to improve the readability of our paper.

Reviewer comment (Novelty):
Theorem 2 should not be called a theorem, as it is a text-book level property in information theory and should not be considered a contribution of the paper.

Author response:
A1: We apologize for any confusion here. Although Theorem 2 is a textbook-level property, to the best of our knowledge, it hasn't been formally presented in any textbooks, which might not be easy to derive or understand for readers unfamiliar with information theory. Considering its importance for deriving the lower bound of the conditional state entropy, we decided to introduce it as a theorem. Following the reviewer's suggestion, we have changed Theorem 2 to Lemma 2 in the revised paper.

Although the property itself is not our contribution, its application in the unsupervised skill discovery domain to derive the lower bound of the conditional state entropy $H(\rm{s}|\rm{z})$ is one of our main contributions. Previous IM methods like APT (Liu & Abbeel 2021) and CIC (Laskin et al. 2022), which directly maximize the state entropy $H(\rm{s})$, exhibit poor performance in state coverage and skill diversity, as shown in Fig. 1 and Table 2. By indirectly maximizing the state entropy $H(\rm{s})=H(\rm{s}|\rm{z})+I(\rm{s};\rm{z})$ through alternating maximization of the lower bounds of $H(\rm{s}|\rm{z})$ and $I(\rm{s};\rm{z})$, our CIM for RFPT significantly outperforms all 15 baseline methods in RFPT tasks. Moreover, our CIM for RFPT is 90x faster than LSD (Park et al. 2021) and CSD (Park et al. 2023) thanks to our novel lower bound of the conditional state entropy for unsupervised skill discovery.

Reviewer comment (Theory):
Equation (5), despite being the main contribution of the paper, is very confusing. The constraint seems to be on the function $\phi$, but the intrinsic objective does not depend on $\phi$. It is unclear whether the constraint should be on the learned skill $z$ so that there exists a representation of state that leads to good alignment. In that case, the constraint should be $\exists \phi, L_a(\phi(s), z) \leq c$, where $c$ is some constant. The definition of CIM needs to be made clear as it is the main contribution of the paper.

Author response:
A2: There might be a misunderstanding. The constraint is indeed on the function $\phi$ and the intrinsic objective does depend on $\phi$. In fact, competence-based IM methods, e.g., APS (Liu & Abbeel 2021a), CIC (Laskin et al. 2022), MADE (Zhao et al. 2022), typically use the notation $H(\rm{s})$ to stand for $H(\phi(\rm{s}))$ since the state entropy is estimated in the state projection space $\mathcal{Z}$ instead of the original state space $\mathcal{S}$ due to the potential high dimensionality of the original state space, such as 378 in Humanoid. For pixel-based tasks, estimating the state entropy directly in the pixel space could be infeasible. We follow this tradition in the IM domain. Specifically, we also estimate the conditional state entropy in the state projection space and use $H(\rm{s}|\rm{z})$ as an abbreviation for $H(\phi(\rm{s})|\rm{z})$. Therefore, the intrinsic objective $H(\rm{s}|\rm{z})$ (i.e., $H(\phi(\rm{s})|\rm{z})$) actually depends on $\phi: \mathcal{S}\to \mathcal{Z}$. We have added detailed explanation for Eqn. (5) to clarify the dependence of $H(\rm{s}|\rm{z})$ on $\phi$ in the revision. Table 3 shows that the learned state representation $\phi(s)$ in the projection space can significantly impact the optimization of $H(\rm{s}|\rm{z})$. Additionally, in the IM domain, $z$ is typically sampled from a pre-defined distribution, such as the uniform distribution $\mathcal{U}^{n_z}(-1,1)$ for continuous skills when there is no prior knowledge ($n_z$ is the skill dimension). Hence, there is no constraint on $z$ typically. Furthermore, given a certain distribition for $z$, our constraint $\phi(s)\in\arg\min L_a(\phi(s),z)$ is equialent to $\exists \phi, L_a(\phi(s),z)\le c$ when setting $c$ to the minimum of $L_a(\phi(s),z)$, i.e., $c = \min_\phi L_a(\phi(s),z)$.

Reviewer comment (Theory):
Right above Equation (10), the paper states that the bound is tight when $\phi(s)$ and $z$ are well aligned. This does not seem correct, because the function $g$ is not invertible, so the $H(g(s) \mid z)$ bound is never tight.

Author response:
A3: Please allow us to clarify that the function $g=\max(\phi(s)^Tz, 0)$ is invertible when the state representation $\phi(s)$ and the skill $z$ are well aligned. The state representation $\phi(s)$ and the skill $z$ being well aligned means that the state representation and the skill are in the same direction, that is, $\phi(s)^Tz=\|\phi(s)\|\|z\|$. Therefore, when the state representation $\phi(s)$ and the skill $z$ are well aligned, $g=\max(\phi(s)^Tz,0)=\max(\|\phi(s)\|\|z\|, 0)=\|\phi(s)\|\|z\|$, which is an invertible function of $\phi(s)$ given a certain $z$. This is because we can recover $\phi(s)$ from $g$, that is, $\phi(s) = \frac{g}{\|z\|}\frac{z}{\|z\|}$ with $\frac{g}{\|z\|}$ being the norm of $\phi(s)$ and $\frac{z}{\|z\|}$ being the direction of $\phi(s)$. Thus, according to Lemma 1 and Theorem 2, the lower bound $H(g(s)|z)$ is tight when the state representation $\phi(s)$ and the skill $z$ are well aligned. We have formalized this argument as Theorem 3 in the revision and provided the proof in Appendix D.

Reviewer comment (Writing):
It is unclear what $k$ refers to in the sentence “the intrinsic reward of CIM for RFPT is then ….”

Author response:
A4: In the sentence "the intrinsic reward of CIM for RFPT is then...", we use $\rm{k}$ to denote the $\rm{k}$−nearest neighbor, that is, in $r_i^\text{CIM}(s)=\|g(s)-g(s)^\rm{k}\|$, $g(s)^\rm{k}$ is the $\rm{k}$-th nearest neighbor of $g(s)$ according to Eqn. (9). To clarify the notation, we distinguished between the $\rm{k}$ and $k$ when we introduced the $\rm{k}$-NN estimator (Eqn. (9)). Specifically, $k$ represents the $k$-th iteration of policy optimization in RL, while $\rm{k}$ is the $\rm{k}$−nearest neighbor in $\rm{k}$-NN estimator. If it would be clearer, we could use $\kappa$ or $K$ for $\rm{k}$−NN estimator in the revision. We have added this description after the sentence "the intrinsic reward of CIM for RFPT is then..." to make it clear.

Reviewer comment (Theory):
In Equation (11), it is unclear how this constraint optimization problem solves the issue of suboptimality. It essentially ensures that the value of the online-policy is non-decreasing if considering the approximation of $\hat R_{k}$ mentioned in the following paragraph, but this does not guarantee that the policy can always be optimized to the true optimal policy.

Author response:
A5: As discussed in Section 3.2, we leverage the Lagrangian method to convert Eqn. (11) to $\min_{\lambda\le 0}\max_{d_\pi} J_e(d_\pi) + \lambda^{-1} J_i(d_\pi)$, where $\lambda$ is updated via SGD. As long as $\lambda^{-1}$ gradually tends to zero, the issue of suboptimality is solved.  Essentially, the agent gradually focuses on maximizing only $J_e(d_\pi)$ instead of $J_e(d_\pi) + J_i(d_\pi)$ in the final stage of training. Our empirical studies showed that $\tau_k^{\text{CIM}}=\min\\{\lambda_k^{-1}, 1\\}$ can gradually decrease from the initial value 1 to a near-zero number (e.g., $1e-4$ in AntUMaze) throughout the training process. To demonstrate how the issue of suboptimality is solved, we have conducted an additional experiment in extra EIM tasks (designed by Mutti et al. 2021), including SparseHalfCheetah (SHC), SparseAnt (SA), SparseHumanoidStandup (SHS) and SpraseGridWorld (SGW). The test-time average episode rewards $J_e(d_\pi)$ are as follows:

| **Scheduler** | **SHC** | **SA** | **SHS** | **SGW** |
|---|---|---|---|---|
|$\tau_k^{\text{C}}$ | 0.27 | 0.74 | 0.18 | 0.01 |
|$\tau_k^{\text{CIM}}$ | **1** | **0.97** | **1** | **1** |

Our adaptive temperature $\tau_k^{\text{CIM}}$ can effectively reduce the bias introduced by the intrinsic objective $J_i(d_\pi)$, i.e., the agent gradually focuses on maximizing only $J_e(d_\pi)$ to exploit the task rewards as $\tau_k^{\text{CIM}}$ is near zero in the final stage of training, instead of being distracted by the intrinsic objective to do superfluous exploration. We have appended the curves of $\tau_k^{\text{CIM}}$ in all EIM tasks in Appendix F.2.

Reviewer comment (Writing):
Could the authors explain what it means by having bias for exploration, a phrase that occurs many times in the Introduction? By bias, we typically mean that some estimator is inconsistent. It is not clear what it means by bias for exploration in this context.

Author response:
A1: In response to your question, it's important to distinguish between two types of biases discussed in the Introduction. The first type is the inductive bias in RFPT tasks, such as the $x-y$ prior utilized by DIAYN (Eysenbach et al. 2018) and DADS (Sharma et al. 2019) for RFPT tasks. This form of inductive bias can be regarded as feature engineering (Park et al. 2021). As LSD (Park et al. 2021) and CSD (Park et al. 2023), we don't rely on feature engineering when designing CIM for RFPT to ensure the generality of our methods.

The second type of bias is the bias introduced by the intrinsic objective in EIM tasks. As outlined in Section 2, in EIM tasks, the agent commonly employs a knowledge- or data-based IM method to stimulate exploration. This is because the task reward is typically sparse and hard to obtain at the start of training. The agent's goal is thus to maximize $J_e(d_\pi) + \tau_k J_i(d_\pi)$ (Eqn. (3)). However, the intrinsic objective $J_i(d_\pi)$ can distract the agent from maximizing the evaluation metric $J_e(d_\pi)$. Specifically, the optimal policy for maximizing $J_e(d_\pi) + \tau_k J_i(d_\pi)$ might not be the optimal policy for maximizing $J_e(d_\pi)$ when the agent is distracted by the intrinsic objective $J_i(d_\pi)$ to collect intrinsic rewards instead of obtaining the extrinsic rewards. Such a distraction phenomenon has also been reported in previous works (Chen et al. 2022). Chen et al. name this bias "intrinsic reward bias". Therefore, we did not use the term "bias for exploration" to describe this type of bias in EIM tasks. Instead, we refer to the "bias" in EIM tasks as the negative influence $J_i(d_\pi)$ exerts on the performance of the agent, the same as Cen et al..

Sorry for the confusion caused by "how to reduce the bias introduced by the intrinsic objective for Exploration with Intrinsic Motivation (EIM)" in the Abstract. We have revised it as "how to reduce the bias introduced by the intrinsic objective in Exploration with Intrinsic Motivation (EIM) tasks".

Reviewer comment (Theory):
The authors mention that knowledge-based and data-based IM introduce non-negligible bias. Why doesn’t skill-based exploration introduce bias? In an EIM setting, it seems that all IM methods with non-decreasing temperature introduce bias.

Author response:
A2: We concur with your perspective that skill-based exploration could potentially introduce bias in EIM tasks. However, as stated in Section 2, to the best of our knowledge, all previous related studies primarily employ knowledge- or data-based exploration methods in EIM tasks, rather than skill-based exploration methods. We thus only mention that knowledge- and data-based IM introduces non-negligible bias in EIM tasks. We hypothesize that one key reason for this is the complexity of simultaneously discovering skills using a skill-based exploration method and maximizing the extrinsic objective $J_e(d_\pi)$ in the sparse-reward EIM tasks. We are willing to explore EIM with skill-based exploration in our future work.

Reviewer comment (Novelty):
Novelty is limited; the algorithm mainly combines two algorithmic approaches in exploration: increasing state coverage and skill discovery.

Author response:
While our algorithm may seem like a combination of data- and competence-based Intrinsic Motivation (IM) methods, the techniques and optimization procedures proposed in this work differ significantly from previous methods. Specifically, we don't directly maximize the state entropy $H(\rm{s})$ as previous data- and competence-based IM methods do. Instead, we propose to maximize a newly introduced lower bound of the conditional state entropy $H(\rm{s}|\rm{z})$, and a novel lower bound of the Mutual Information (MI) $I(\rm{s};\rm{z})$. We argue that this approach is novel and could efficiently enhance state coverage $H(\rm{s})=H(\rm{s}|\rm{z})+I(\rm{s};\rm{z})$ and promote dynamic and diverse skill discovery. Empirically, CIM for RFPT improves the performance across all standard RFPT evaluation metrics, including skill diversity (as shown in Fig. 1), state coverage (as shown in Table 2), pre-training sample efficiency (90x faster than leading methods LSD and CSD), and fine-tuning efficiency (on par with top-tier methods such as CIC, BeCL and MOSS as shown in Table 4).

Reviewer comment (Novelty):
I’m not sure the knowledge-based and data-based classification of methods in the introduction is entirely accurate. For example, the objective given in Zhang et al. 2021 can recover maximum entropy exploration technique in special case.

Author response:
The classification of methods into knowledge-based and data-based, as introduced, is widely recognized in previous works like APT (Liu & Abbeel 2021b), APS (Liu & Abbeel 2021a), URLB (Laskin et al. 2021)), MADE (Zhang et al. 2021), CIC (Laskin et al. 2023), and MOSS (Zhao et al. 2022). Regarding the paper MADE (Zhang et al. 2021) the reviewer mentioned, Zhang et al. proposed two regularizers. One is the general regularizer $R(d_\pi)$, which is equivalent to our intrinsic objective $J_i(d_\pi)$. The other is the specific MADE regularizer $E_{\rm{s}} [ (\rho_\pi^{-1}(s)d_\pi^{-1}(s))^{1/2} ]$, which belongs to knowledge-based IM methods. Thus, when we refer to the knowledge-based IM method MADE, we mean this specific MADE regularizer. Zhang et al.'s regularizer perspective aligns with our intrinsic objective perspective. Actually, all exploration techniques can be recovered with a specific intrinsic objective $J_i(d_\pi)$, as shown in the second column of Table 1. For instance, maximum entropy exploration like MaxEnt and APT can be recovered by setting $J_i(d_\pi)=H(s)=E_{\rm{s}}[-\log d_\pi(s)]$.

Reviewer comment (Theory):
$\tau_k$ is more like a hyperparameter to control the trade-off between exploration and exploitation and is not really a temperature parameter—those are usually used in the form $\exp(\tau x)$ such as in softmax outputs or Boltzmann exploration.

Author response:
We agree with the reviewer that $\tau_k$ can also be regarded as a hyperparameter. We refer to $\tau_k$ as the temperature to maintain consistency with previous works like MADE (Zhang et al. 2021). In MADE, Zhang et al. proposed the regularized objective $J_e(d_\pi) + \tau_k R(d_\pi)$. Here, the former represents the extrinsic objective, and the latter formulates the intrinsic objective. We unify such a regularized objective in Intrinsic Motivation Reinforcement Learning (IMRL). Specifically, it is equivalent to the objective of Extrinsic Intrinsic Motivation (EIM) tasks $J_e(d_\pi) + \tau_k J_i(d_\pi)$. Therefore, we name $\tau_k$ as a temperature parameter, which determines the strength of exploration. If it would be clearer, we could revise "tempareture parameter" as "hyperparameter" as the reviewer suggests.