Paper: Hierarchical Empowerment: Towards Tractable Empowerment-Based Skill Learning (ICLR-2024)

Reviewer comment (Experiments):
Another major weakness of this work is its limited empirical evaluation. For me, the proposed method is much closer to previous unsupervised GCRL methods (e.g., Skew-Fit, EDL, LEXA, etc.) than empowerment-based methods (e.g., DIAYN, DADS, HIDIO, etc.) given that (1) the latter uses a separate latent skill space $Z$ that is not "grounded" to the state space and (2) the former has the two-stage structure of goal proposal and goal reaching, similarly to the proposed approach. However, the authors only compare their method with empowerment-based approaches, which have already been known to struggle to reach distant goals (as shown in multiple prior works, such as EDL and LSD). I believe comparisons with unsupervised GCRL methods are necessary to assess the effectiveness of this method.

Author response:
Thank you for your feedback.

### Missing comparisons to Unsupervised Goal-Conditioned Algorithms (MEGA, Skew-fit, EDL, etc.)

We disagree that a comparison with the unsupervised GCRL algorithms (MEGA, Skew-fit, EDL) is necessary.  Given the importance of Empowerment as a way to find the space of reachable states and thus quantify the size of an agent’s skillset, there needs to be algorithms that optimize the full Empowerment objective.  But MEGA, Skew-fit, and EDL do not optimize the full Empowerment objective.  Rather they optimize two different objectives.  In one process, they optimize $H(Z|s_0)$ or $H(S_N|s_0)$ (e.g., training an autoencoder on visited states to obtain a goal space $p(z|s_0)$).  At the same time, but in a second separate process, they train and goal-conditioned policy using goals sampled from the goal space to minimize $H(Z|s_0,S_N)$ or $H(S_N|s_0,Z)$.  The clear failure case for this approach is when there is randomness and the goal states from the goal space are not consistently achievable.  In this situation, this strategy may not calculate empowerment accurately (i.e., find the largest space of reachable states.) because the skill space is not optimized to learn skills that are consistently achievable. Thus in our baselines, we just considered algorithms that optimize the full Empowerment objective.

Reviewer comment (Theory):
The proposed method seems to assume access to the ground-truth transition dynamics function, which prevents its applicability to general environments.

Author response:
### Where is the model used and where is it required?

We use the model to help generate transitions for all the goal-conditioned and goal-space policies.  The goal-conditioned policies need unbiased (prior state,action,next state) tuples and the goal-space policies need unbiased (skill start state, goal space action, epsilon, skill end state) tuples.  The model is not required for the base level goal-conditioned policy because a replay buffer of (prior state, action, next state) transitions would be unbiased.  A model is also technically not required for the goal space policies because the Q function from the goal-conditioned policies at the skill start states could be used as the critic for the goal space policy but we chose to use the model to generate transitions to train a separate goal space critic.

Reviewer comment (Writing):
In terms of writing, I feel Section 3.1 is not sufficiently clear.

Author response:
### Section 3.1 Clarifications

$g$ represents the reparameterization trick.  For our uniform goal space distributions, each dimension of $z_i = -\text{halfwidth (output by goal space policy)} + 2*\text{halfwidth}*\text{eps}$, in which eps is a sample from a unit uniform random variable.  $h$ is just a projection to the goal space (e.g., the (x,y) coordinates from the state).

The line "instead of sampling the skill z from the fixed variational distribution" was just referring to how the log of the variational posterior $\log q(z|s_0,s_N)$ is calculated, not how the skill $z$ is sampled.  Instead of computing $\log N(z; \text{mean}=s_N,\text{std}=\sigma_0)$ we use  $\log N(z + h(s_0);\text{mean}=s_N,\text{std}=\sigma_0)$.  This causes the skill $z$ to reflect desired change in state $s_0$ (e.g., desired change in the x and y coordinates from the skill start state $s_0$) and not actual actual states.  This is helpful because it initially centers the learned goal space on the start state $s_0$.

Reviewer comment (Theory):
"However, the existing empowerment-based skill-learning approaches that learn the variational distribution $q_\psi(z | s_0, s_n)$ also require a model in large domains to obtain unbiased gradients of the maximum likelihood objective.": I'm not sure if this can justify the second limitation because (1) albeit biased, previous empowerment-based approaches can still work without a model and (2) we can relabel $z$ as in P-HER (Choi et al., 2021) to make it (much) less biased with respect to the current skill discriminator. In this regard, I think "First, similar to other empowerment-based skill-learning methods, our framework assumes the agent has access to a model of the environment’s transition dynamics." is a bit misleading.

Author response:
### “Albeit biased, previous empowerment-based approaches can work without a model.”

It’s difficult to compare our empowerment-based work with earlier results because both our (a) skill space learned for each skill start state and (b) the distribution of skill start states seem to be vastly larger than prior work (i.e., our $p(z|s_0)$ and $p(s_0)$ are significantly more entropic), particularly in our larger settings.  Storing and sampling biased $(s_0,z,s_N)$ tuples from a replay buffer seems unlikely to scale to very large domains.  For instance, given that the replay buffer contains stale (i.e, biased) data, you can only use the recent past.  But then in large domains, much of the state space may not be continually visited so if you cut off your replay buffer, then your empowerment optimization may be missing those reachable states.

Reviewer comment (Experiments):
One of the contributions of the method is to learn multiple levels of hierarchy (previously investigated in [3], which is not cited), but there are no experiments that separate the effect of the empowerment objective from the multiple levels. It is unclear which of these provides the performance gains.

Author response:
We may be misinterpreting the question, but we are not sure how to separately assess them because we are not aware of other ways to build a goal-conditioned hierarchy in an unsupervised manner.  Our work shows that empowerment can be a way to generate each level of the goal-conditioned hierarchy in an unsupervised manner.

Reviewer comment (Theory):
The paper states "The benchmarks either (i) do not provide access to a model of the transition dynamics." The necessity for the model of the transition dynamics is not explained.

Author response:
The primary need for the model is to generate unbiased (state, subgoal action, next state) tuples for training the above-base level goal-conditioned policies with RL.  Given that the goal-conditioned policies at each level are continually changing, storing and sampling these transitions from a replay buffer would provide biased samples.  

In addition, we have tried to argue that existing empowerment-based skill-learning methods (e.g., DIAYN) that train the posterior distribution $q_{\phi}(z|s_0,s_N)$ with maximum likelihood will also need a model when the distribution of skill-start states $p(s_0)$ is highly entropic.  Maximum likelihood training of $q_{\phi}(z|s_0,s_N)$ requires on-policy tuples (s_0,z,s_N), which require executing the _current_ skill-conditioned policy $\pi_{\theta}(a|s,z)$ from the skill start state $s_0$ for $N$ actions.  Without a model, this would require executing the skill-conditioned policy for every skill $z$ at every start state $s_0$ to get unbiased (s_0,z,s_N) tuples.  This is not practical when p(s_0) and p(z|s_0) are highly entropic (i.e., when you need to learn a large set of skills from each state in a large set of states).  The alternative of storing the (s_0,z,s_N) tuples (e.g., p-HER [1]) in a replay buffer will provide biased samples because the skill-conditioned policy is continually changing.  Having a model would enable the user to execute the skill-conditioned policy from the necessary (s_0, z) combinations in parallel.

Reviewer comment (Theory):
The paper claims "The entropy term H ϕ(Z | s 0) encourages the goal space policy µ ϕ to output larger goal spaces." This seems incorrect, as the goal space is predefined. Perhaps the intended meaning is that the distribution of generated goals will cover more space.

Author response:
The output of the goal space policy includes the half-lengths of each dimension of the uniform goal space distribution.  The entropy term $H_{\phi}(Z|s_0)$ in the Goal-Conditioned Empowerment objective incentivizes the goal space policy to output larger halfwidths, which is what we meant by “larger goal spaces.”  The dimensionality of the goal space is fixed and we can clarify that in the paper.

Reviewer comment (Theory):
The claim that "hand-crafted goal space may only achieve a loose lower bound on empowerment" is made with little justification.

Author response:
We will break this down into two possible cases.  In the simpler case of the human trainer setting too small of a goal space (i.e., there are additional goals outside the hand-crafted goal space that can be achieved), $H(S_N|s_0)$ (the upper bound of empowerment assuming a discrete goal space) will be smaller than when the goal space includes all of the achievable goal states.  This would produce a loose lower bound on empowerment if the hand-crafted goal space is significantly smaller than the achievable goal space. 

In the case of the human trainer setting the goal space too large (i.e., there are goals in the goal space that cannot be achieved in $N$ steps), then the goals that cannot be achieved will yield skill end states $s_N$ that are redundant as they can be achieved by other goals (e.g., the goal $s_N$).  This will lower $H(S_N|s_0)$ unless all the redundant skill end states $s_N$ are distributed uniformly over the reachable state space $S_N$.  If the redundant states are not uniformly distributed across $S_N$, this again can produce a loose lower bound on empowerment. 

In our case of the continuous goal space distributions, it’s more complicated because there is no upper bound on the conditional entropy $H(S_N|s_0,Z)$, which can reach infinity.  But clearly in practice there will be situations when the above two cases apply and the empowerment that is achieved is significantly lower than what is possible with a learned goal space.

Reviewer comment (Experiments):
The authors did not provide enough discussion on how to decide on k and how to design the structure. It will be great if the authors can provide explanations and more insight on this.

Author response:
Thank you for your feedback.  

We agree with your characterization of the strengths and weaknesses of the algorithm.  In regards to selecting the number of levels $k$, our main criteria was that we wanted each level to learn a short sequence of actions given the difficulties of credit assignment in RL.  Given that the $k$-th root of the number of actions in an episode, (i.e., $(\text{actions per episode})^{\frac{1}{k}}$), reflects the lengths of the policies each level needs to learn, we were looking for the $k$-th root to be some small number (ideally under 10).  Although increasing the number of levels can make learning long horizon skills more tractable, the main drawback is that it increases the wall clock time of training increases.  Training the top level goal-conditioned and goal-space policies in the hierarchy requires simulating $N^k$ primitive actions, which increases exponentially with $k$.   $N$ here refers to the maximum number of actions executed by each level.