Paper: Piecewise Linear Parametrization of Policies: Towards Interpretable Deep Reinforcement Learning (ICLR-2024)

Reviewer comment (Novelty):
The paper overlooks a whole body of work on programmatic policies, particularly the Programmatically Interpretable RL (PiRL) line of research as seen in Verma et al., which aims to create interpretable policies for RL problems. Key related works include Verma et al. and Bastani et al. using imitation learning, Qiu and Zhu proposing a differentiable approach with architectures resembling oblique decision trees, and further relevant methods from Lee and Jaakkola and from Orfanos and Lelis. The omission of these works limits the paper’s ability to properly place its contributions within existing literature.

Author response:
### Relationship to programmatic policies
We thank the reviewer for the references that they provided us.

We agree that Programmatically interpretable RL is an area relevant to our research, and we updated our manuscript to include the references in our related works. We note that there are several significant differences with our approach. As you mention, both the papers from Bastani et al. and Verma et al. use imitation learning, while we directly train an interpretable policy by interaction, from scratch, setting our work within functional interpretability rather than distillation (we quickly touched on the distinction in Appendix A). 

The article from Qiu and Zhu is very interesting and closer to our work. A major difference with our work is that in most of their control experiments from Section 3 and 4, Qiu and Zhu’s algorithm assumes access to pre-trained abstract policies performing abstract actions such as going straight or turning (which they call ensemble policies). Their algorithm then learns to combine them for high level planning. They do mention affine policies in Section 2., but show that they perform much worse in Table 2. On the contrary, HC learns jointly the affine sub-policies (low-level control) and the high-level planning, from scratch. HC manages to solve complex tasks despite needing to discover by itself what movements are useful to achieve its goal. Moreover, as you mention, their policy is ultimately learned as an oblique decision tree where the leaves execute pre-trained abstract actions (whereas ours could be assimilated to an oblique decision tree where the leaves execute linear sub-policies), but as far as we understand, they do not use a neural network to model this tree (which could lead to additional constraints, such as having to keep the actor small (Orfanos 2023)).

The work from Orfanos and Lelis was submitted to arXiv past the “contemporaneous” deadline of ICLR, but we are happy to discuss it now and include it in the literature review. Thank you for mentioning it to us. The authors do use a neural network actor in an actor critic setting, similarly to us. They then translate it into a programmable policy expressed as an oblique decision tree. They however have to restrict themselves to small actor architectures to prevent their tree from becoming uninterpretable, as mentioned in their introduction or experiments section. We are not bound by such a restriction, since we directly restrict the number of sub-policies that HC can exhibit while maintaining the complex partition induced by large neural networks. Moreover, we evaluate HC in both control and navigation settings, on the contrary to this paper. 

Overall, PiRL algorithms focus on the structure of the policies, expressing them for instance as finite state machines or small decision trees. In the latter case, the number of possible different behaviors of the policy (one per leaf) are implicitly defined by the height of the tree. The leaves apply abstract, discrete actions (Trivedi 2021, Qiu & Zhu 2022, Liu & Hu 2023) in navigation settings, or linear / PID sub-policies in control settings. In comparison, HC learns a small, pre-determined fixed number of linear sub-policies, while keeping the structure of the policy highly flexible. As a consequence, HC learns to perform both low-level control and high-level planning at the same time.

Reviewer comment (Novelty):
Several claimed distinguishing properties of the HC architecture in the paper are also achievable with ReLU networks:  
(1) Making functions \(a\) and \(\theta\) explicit is possible with ReLU networks, where \(a\) is the set of weights in hidden layers and \(\theta\) is the function learned when the activation pattern is fixed.  
(2) Controlling the number of unique sub-policies via the Gumbel-Softmax layer dimension is analogous to controlling the number of neurons or using L1 regularization in ReLU networks.  
(3) Discontinuity at borders between linear regions exists in ReLU networks as they also switch linear prediction functions between regions.  
(4) Local interpretability for individual examples matches that achievable when mapping ReLU networks to Oblique Decision Trees.  
These similarities raise the question of what truly differentiates HC from small ReLU networks.

Author response:
### Relationship of HC with ReLU networks
We agree that HC is indeed extremely linked with MLP-ReLU networks (abbreviated as ReLU networks in the following), as the latter is actually the starting point from our architecture (first paragraph of Sec 3.1). However, we made some strong changes to the ReLU network architecture in order to achieve desirable interpretability properties which we identified at the end of Sec. 2 and beginning of Sec. 3.
Overall, the differences between ReLU networks and HC are best exemplified by Fig. 1 (right):
ReLU networks induce (1) a complex partition of the input space and (2) a different linear sub-function in each linear region (one color per linear sub-function).
HC also induces (1) a complex partition (useful to solve hard tasks from scratch), but (2) only exhibits a small number of linear sub-functions (illustrated by using only 4 colors, hence 4 linear sub-functions). This is in stark contrast with ReLU networks of all but the smallest architecture.

In addition, we have actually compared ReLU networks (including with small architectures) with HC in our submission:
In Appendix B, Fig. 6, we show evidence that HC reaches a better performance-interpretability trade-off than ReLU networks for a wide range of architectures. This leads HC to a high performance while using a very small number of linear sub-policies.
In Appendix C, Fig. 7, we show how a ReLU network uses essentially a new sub-policy at each timestep, on the contrary to HC.

We now answer your points detailing the differences between HC and ReLU networks, following your points:

### 1. Explicit formulation of $a$ and $\theta$

We agree that it is possible to recover $a$ and $\theta$ for ReLU networks, as we indicated in the beginning of Sec 3.1. However, these functions do not appear explicitly in the computation of the output $f(x)$, and need to be recovered (in the case of the local linear coefficients, by taking the input Jacobian of $f$).
On the contrary, we make $a$ and $\theta$ explicit within the computation made by the neural net, by composing a Gumbel network $a$ and the local linear coefficients $\theta$. The explicit modeling of $a$ within the forward pass of the actor lets us set the *exact number* of sub-policies as a hyperparameter, unlike ReLU networks (we expand on this last statement in the answer to your second point).

Following your comment, we modified the manuscript (Sec 3.1) to clarify that $a$ and $\theta$ now appear explicitly in the computation (in purple).

## Rebuttal 2 (ID: YxgiG5xC2J)
### 2. Controlling the number of sub-policies
It is possible to upper bound the number of sub-policies of ReLU networks through their architecture. However, the exact number of sub-policies depends on the activation patterns that are learnt (a network with only parameters with value 0 has only one linear region). Therefore, there is no hyperparameter setting directly how many linear sub-policies the network can use in ReLU networks, on the contrary to HC, where we can set this hyperparameter (the output size of the Gumbel network, $d_G$).

Another issue with ReLU networks is that the number of linear regions can grow very quickly with the width and depth of the network. While it is only a theoretical upper-bound, this is illustrated in (Montufar, 2014). Some more recent work has shown evidence that the actual number of linear regions used by the network grows less quickly (Hanin & Rolnick, 2019). Therefore, we performed in Appendix B an extensive empirical study to evaluate, in the DM control benchmark, how many unique sub-policies a ReLU network actually uses. We did this for a wide range of depth and width of ReLU networks.

The results are illustrated in a Pareto plot, in Fig. 6. We analyze in detail these results in Appendix B, both theoretically (B.1) and empirically (B.2). In summary, HC obtains a great performance with far fewer sub-policies used, because we are able to constrain this number explicitly with $d_G$. In comparison, ReLU networks use far more sub-policies to reach a comparable performance.
We also show in Fig. 7 (Appendix C) the cumulative number of unique sub-policies re-used over a trajectory, which shows that ReLU networks use essentially a new linear sub-policy at almost every timestep (950/1000). 

Finally, we note that while one could add an L1 penalty on the parameters of the neural network to hopefully limit the number of linear regions used, the exact number of sub-policies expressed by the network would not be known a priori. On the contrary, HC lets us set before training the strict number of linear sub-policies that we are willing to allocate to the problem.
Still, following your advice, we looked more carefully if adding an L1 penalty enabled a higher performance while keeping the number of linear sub-policies used in a trajectory low, extending the analysis of Appendix B.
We added the results in a new Appendix B.3 section, which we briefly comment now.
On our benchmark, we did not find that adding an L1 penalty significantly improved the performance/interpretability trade-off. We tried the value mentioned in the Orfanos paper, as well as more general ones. The penalized policies were either using a very small number of linear sub-policies for a poor performance, or a high number for a high performance. In comparison, HC uses few linear sub-policies while reaching a high performance.

### 3. Continuity
Thank you for your comment. We mean to say that HC usually presents a discontinuity at the border between linear regions, while ReLU networks do not because they express continuous functions (and are therefore continuous at the border between linear regions, even though they are indeed usually not differentiable there). HC is in general neither continuous nor differentiable at the border between linear regions. 

We included this difference in the class of functions represented for exhaustivity, and did not intend to present this as an advantage of HC over ReLU networks. We verified that the non-continuity of HC was not an issue for the stability of the policy (Appendix D.6).

### 4. Local interpretability
Thank you for your remark. We agree that shallow (oblique) decision trees are also locally interpretable. However, deeper trees tend to weaken this property, as each leaf receives less examples, meaning that the coefficients of the linear function that is applied to a leaf are only applied to a few examples. Therefore, to ensure that the linear coefficients apply to a sufficient number of inputs, one solution is to regularize the depth of the tree.

Similarly, all but the smallest ReLU networks lead in general to a high number of sub-policies used in practice (Appendix B and C), which significantly weakens their local interpretability.
On the other hand, we achieve with HC a similar effect to shallow decision trees by explicitly limiting the number of linear sub-policies that can be applied to a given example to only a small number. This forces the linear coefficients to be re-used for several inputs, essentially ensuring that the explanations provided by the coefficients are valid for a wide number of inputs.

Reviewer comment (Experiments):
The experiments in the paper do not include small ReLU network baselines, despite the similarities to HC and evidence from Qiu and Zhu that such architectures perform well on the same benchmarks. Including these baselines would provide a fairer and more informative comparison.

Author response:
Dear reviewer,

Thank you for your answer and for your suggestions, which we followed over the past few days.

Overall, we performed the two proposed experiments comparing (1) HC to the $\pi$-PRL policy from Qiu and Zhu and (2) HC to a distilled small actor. We found that both methods did not lead to a successful policy in Cheetah run, on the contrary to HC. We have updated the appendix with two detailed new sections to document our results in the comparison with programmatic (Appendix D.8, Fig. 21+22) and distillation (Appendix B.4, Fig. 7) policies. We now summarize our findings.

### Comparison to programmatic policies
Following your suggestion, we compared HyperCombinator with the $\pi$-PRL policy from Qiu and Zhu (2022).
We dedicated quite some time to understand and modify the code they provided online, which uses TRPO as the algorithm to train their policy, to be compatible with affine leaves and the Deepmind control suite. We applied their technique to the Cheetah run task, where we used affine controllers in each leaf of the tree. We tested two versions of the algorithm, one producing soft decision trees (using Eq. 1 of their paper) and one using hard decisions at each branching, in order to obtain a hard, regular decision tree (reflecting Fig. 3 of their paper). We also trained a TRPO algorithm with a neural policy (two layers of 256 units) in order to provide a comparable baseline to $\pi$-PRL.
We illustrate the results in Fig. 21 and 22 and comment on them, along with the experiment details, in Appendix. D.8. We also analyzed the behavior of the best performing seed, given that the results had high variance.

In summary, we found that $\pi$-PRL did not match the performance of the baseline TRPO, which itself performs well below HC in this setting. 

### Comparison to distilled policies
In the second experiment, we compared as suggested the HyperCombinator with a student policy that was trained to imitate an expert teacher policy. 
We trained the student (a one-hidden layer network with 32 hidden units, as the network of Orfanas et al (2023)) to match the predictions of the teacher (a SAC agent with two hidden layers of 1024 units trained to convergence) by minimizing the mean squared error between their predicted actions. We used as dataset the states encountered by the teacher during its training run. Each state in the dataset is then labeled with the prediction of the fully trained teacher. We tried a variety of batch sizes and considered regularizing the weights of the teacher, similarly to Orfanas et al (2023).

We illustrate the results in Fig. 7 of the manuscript. Despite a successful minimization of the mean squared error (Fig. 8, log scale), the students fail to obtain a satisfactory performance at Cheetah run.
We think the main reason for this catastrophic failure of the students is due to a mismatch between the objective they are trained towards (minimization of the action prediction error between the teacher and the student) and the purpose of the policies (maximize the return in Cheetah run). Because of their considerably lower capacity, students necessarily make approximations in order to fit the teacher. Since we cannot control in which states these approximation errors are made, it is possible for a student to poorly approximate a key state, such that the student is thrown off its trajectory when rolled out in the environment, even though it obtained a good mean squared error overall.
We think this issue is a key problem of distillation, as corroborated by Qiu and Zhu (2022)  (Section 6). We explored in Fig. 9 a possible fix, which is to include inputs seen by the student in the training data (Verma, 2018). This did not improve the performance of the student.

On the other hand, the HyperCombinator is trained with reinforcement learning directly on the task. Therefore, it does not suffer from this mismatch and directly optimizes the desired objective (the cumulative sum of rewards), and as a consequence manages to learn a small policy that performs very well. 

 _

We hope that these two additional comparisons will convince you of the value of our HyperCombinator approach in learning actors that interact with very few sub-policies with the environment, while reaching high performance.

Reviewer comment (Novelty):
Additional related works on interpretable RL policies, such as those by Inala et al., Koul et al., Aleixo and Lelis, Trivedi et al., and Liu et al., are relevant to this submission and could help position the contribution within the broader literature on interpretable and programmatic policies. While perhaps less crucial than PiRL, they still provide context.

Author response:
### Relationship to programmatic policies
We thank the reviewer for the references that they provided us.

We agree that Programmatically interpretable RL is an area relevant to our research, and we updated our manuscript to include the references in our related works. We note that there are several significant differences with our approach. As you mention, both the papers from Bastani et al. and Verma et al. use imitation learning, while we directly train an interpretable policy by interaction, from scratch, setting our work within functional interpretability rather than distillation (we quickly touched on the distinction in Appendix A). 

The article from Qiu and Zhu is very interesting and closer to our work. A major difference with our work is that in most of their control experiments from Section 3 and 4, Qiu and Zhu’s algorithm assumes access to pre-trained abstract policies performing abstract actions such as going straight or turning (which they call ensemble policies). Their algorithm then learns to combine them for high level planning. They do mention affine policies in Section 2., but show that they perform much worse in Table 2. On the contrary, HC learns jointly the affine sub-policies (low-level control) and the high-level planning, from scratch. HC manages to solve complex tasks despite needing to discover by itself what movements are useful to achieve its goal. Moreover, as you mention, their policy is ultimately learned as an oblique decision tree where the leaves execute pre-trained abstract actions (whereas ours could be assimilated to an oblique decision tree where the leaves execute linear sub-policies), but as far as we understand, they do not use a neural network to model this tree (which could lead to additional constraints, such as having to keep the actor small (Orfanos 2023)).

The work from Orfanos and Lelis was submitted to arXiv past the “contemporaneous” deadline of ICLR, but we are happy to discuss it now and include it in the literature review. Thank you for mentioning it to us. The authors do use a neural network actor in an actor critic setting, similarly to us. They then translate it into a programmable policy expressed as an oblique decision tree. They however have to restrict themselves to small actor architectures to prevent their tree from becoming uninterpretable, as mentioned in their introduction or experiments section. We are not bound by such a restriction, since we directly restrict the number of sub-policies that HC can exhibit while maintaining the complex partition induced by large neural networks. Moreover, we evaluate HC in both control and navigation settings, on the contrary to this paper. 

Overall, PiRL algorithms focus on the structure of the policies, expressing them for instance as finite state machines or small decision trees. In the latter case, the number of possible different behaviors of the policy (one per leaf) are implicitly defined by the height of the tree. The leaves apply abstract, discrete actions (Trivedi 2021, Qiu & Zhu 2022, Liu & Hu 2023) in navigation settings, or linear / PID sub-policies in control settings. In comparison, HC learns a small, pre-determined fixed number of linear sub-policies, while keeping the structure of the policy highly flexible. As a consequence, HC learns to perform both low-level control and high-level planning at the same time.

Reviewer comment (Evaluation):
The evaluation of interpretability in the paper is weak, relying mainly on anecdotal evidence and illustrative plots of sub-policies. While these examples are appreciated, they do not amount to a robust or systematic evaluation.

Author response:
### About the interpretability of our method
We thank the reviewer for their question. As a main advantage, HC brings transparency to the decision process through the two levels of interpretability: the linear coefficients of the very few sub-policies, and the sequence of chosen sub-policies over a trajectory. 

This transparency is a tool that lets stakeholders get insights about what their policy is doing, and how it is interacting with the environment. We argue the added transparency improves the interpretability over regular ReLU networks, which model far more linear sub-policies (Appendix B and C). At the same time, HC’s performance does not collapse for very difficult mazes, provided that some interpretability is traded-off for performance by increasing the number of linear sub-policies.

For instance, we think the transparency of HC gives domain experts (who have knowledge of the environment) a useful tool to understand the policy they work with. We illustrate this aspect with a small analysis in Appendix D.7. We briefly analyze the meaning of the coefficients of the linear sub-policies and show how to interpret them in light of the sequence of sub-policies chosen.

Finally, we agree with the reviewer that examples remain subjective since the interpretability depends on the stakeholder. Simultaneously, we designed HC such that the stakeholder could decide for themselves the level of simplicity of the policy, by setting the number of sub-policies $d_G$.
Therefore, we think HC is a valuable step in the direction of inherently interpretable policies that solve complex tasks.

Reviewer comment (Evaluation):
The interpretability literature, in general, is weak on formal evaluation, but the emphasis on interpretability in this paper makes the lack of a clear, rigorous assessment more problematic. The key unanswered question remains: are these policies really interpretable, and if so, to whom?

Author response:
### About the interpretability of our method
We thank the reviewer for their question. As a main advantage, HC brings transparency to the decision process through the two levels of interpretability: the linear coefficients of the very few sub-policies, and the sequence of chosen sub-policies over a trajectory. 

This transparency is a tool that lets stakeholders get insights about what their policy is doing, and how it is interacting with the environment. We argue the added transparency improves the interpretability over regular ReLU networks, which model far more linear sub-policies (Appendix B and C). At the same time, HC’s performance does not collapse for very difficult mazes, provided that some interpretability is traded-off for performance by increasing the number of linear sub-policies.

For instance, we think the transparency of HC gives domain experts (who have knowledge of the environment) a useful tool to understand the policy they work with. We illustrate this aspect with a small analysis in Appendix D.7. We briefly analyze the meaning of the coefficients of the linear sub-policies and show how to interpret them in light of the sequence of sub-policies chosen.

Finally, we agree with the reviewer that examples remain subjective since the interpretability depends on the stakeholder. Simultaneously, we designed HC such that the stakeholder could decide for themselves the level of simplicity of the policy, by setting the number of sub-policies $d_G$.
Therefore, we think HC is a valuable step in the direction of inherently interpretable policies that solve complex tasks.

Reviewer comment (Theory):
The method has limitations that prevent it from being applied to complex problems of larger scale: The policy loses interpretable as soon as the number of linear sub-policies grows large.

Author response:
We thank the reviewer for their feedback. We are glad they enjoyed reading the paper.
HC indeed does not escape the performance-interpretability trade-off, and increasing the number of sub-policies tends to improve performance but reduce interpretability (Fig. 4). We think one of the interesting aspects of our work is that the stakeholder can decide this trade-off for themselves, by fixing how many sub-policies they are willing to use.

Reviewer comment (Experiments):
In the Cheetah control setup, is the action space continuous? If so, what does the "action 0" in Figure 3 mean? Is it one dimension of the action? Then it is not proper to refer to it as "for one action of $\tilde{\pi}$".

Author response:
In Fig. 3, you are absolutely right, we are referring to the computation of the logits for the dimension 0 of the action space. We have updated the legend accordingly (in red), thanks.

Reviewer comment (Experiments):
Are subpolicies trainable? Besides the diversifying regularizations for the MLP, would similar regularizations encouraging a diversified sub-policy help with the learning and model capacity?

Author response:
The sub-policies are indeed trainable. We learned each sub-policy (jointly with how to choose them) in each of our experiments. We actually tried to diversify the initialization of each sub-policy earlier in the project, but we did not find any benefits in terms of performance at that time. We therefore find the reviewer’s proposition of diversifying the sub-policies themselves (in addition to how to choose them) could be very interesting to revisit in the future, as it might help each sub-policy to specialize even more.

Reviewer comment (Experiments):
There appears to be clear limitations of the method performance wise, as it seems to really struggle to latch onto the similar abilities of the black-box. This can be problematic, as reducing performance has catastrophic effects on user trust (and then appropriate reliance etc.). Or at least, HC64 etc. is needed to get reasonable performance as opposed to HC8.

Author response:
We agree with the reviewer that performance is an important factor in user trust. One of HC’s main advantages is that it lets the stakeholder tune the performance-interpretability trade-off through the number of sub-policies. In certain safety-critical environments or when the stakeholder is not a domain expert, the stakeholder might be willing to sacrifice some of the performance in order to gain insights about how the policy interacts with its environment. 

On the other hand, a domain expert might be comfortable with more complexity, unlocking higher performance. This is illustrated in Fig. 4: HC64 never catastrophically fails, despite the high difficulty of the maze environments and its relative simplicity compared to the MLP baselines. HC8 struggles to solve the harder tasks but leads to simpler visualizations (e.g., Fig. 5). We think this is especially encouraging given the high difficulty of the mazes. 
We update our manuscript in Sec 3.2 to express this point more clearly, in blue.

Finally, some applications demand first and foremost the highest level of performance. We think our approach is outside of this scope, since a performance cost has to be paid for the gains in interpretability.

Reviewer comment (Experiments):
The authors don't actually show the method being particularly useful for anything. Usually, explainability methods are used to debug, teach, regulate, calibrate reliance, or offer recourse, but this method isn't shown to be able to do any of those things.

Author response:
We thank the reviewer for their remark. In this work, our goal was to study the properties that could improve the interpretability of piecewise-linear policies, and to propose a parametrization that let us achieve as many of these properties as possible. Therefore, we studied and evaluated the HyperCombinator as a general-purpose architecture.

The two levels of interpretability of HC relate to its transparency. The linear coefficients inform of the importance of a feature in the computation made by a subpolicy, and the sequence of chosen behaviors gives insights about the structure of the inner decision process of the agent.
In turn, a domain expert can use this transparency to better understand the policy that they roll out (Appendix D.7 gives a low-dimensional example of such an analysis).

The sub-policy sequence visualization could also be used to detect issues at run-time, especially if no video of the agent is available. In Fig. 15 (Appendix D.6), we studied the robustness of HC to perturbations, which we could model as an adverse event. The perturbations that the agent is subjected to are translated into an unusual sub-policy sequence between timesteps 100 and 200. Hence, we could imagine a system that detects a change in the subpolicy sequence, which would in turn raise an alarm that an adverse event has been detected.

Overall, HC opens the gate to new analyses thanks to its transparency, which we think is a significant improvement from the status quo.

Reviewer comment (Writing):
There is a misspelling in the title of parameterization.

Author response:
- Title: we believe that both “parameterization” and “parametrization” are valid ways to spell the word.

Reviewer comment (Experiments):
Section 3 is limited to just "counterfactual reasoning", whereas there are many other types of contrastive explanation which the method could also generate.

Author response:
- Contrastive explanations: we thank the reviewer for their suggestion. We agree that the architecture of HC opens up new possibilities, such as analyzing why a given sub-policy was chosen rather than another.

Reviewer comment (Writing):
The reviewer does not fully understand (P2) here.

Author response:
- P2: we clarified the meaning of the property in p.3 following your suggestion.

Reviewer comment (Writing):
On p6, the reviewer suggests not saying "remarkable" and letting the readers decide themselves.

Author response:
- We reformulated the “remarkable” mention in p.6 following your suggestion.

Reviewer comment (Evaluation):
It would be good to briefly elaborate in Fig2 why Finger spin behaves differently.

Author response:
- Finger spin consists in the very quick repetition of the same precise movement in order to maximize the reward. Policies with more degrees of freedom have an advantage here as they can reach a better precision thanks to their higher expressivity. We find this not too surprising, as complex policies end up performing better than HC given enough samples (Fig. 12, Appendix D.4). We added a remark about this phenomenon in Appendix D.4.

Reviewer comment (Experiments):
It is unclear whether this work applies to pixel-based input data.

Author response:
It is indeed entirely possible to use interpretable features while conditioning the Gumbel network on pixel data. For instance, in DM control, we could define the observations as multimodal: one pixel representation of the environment, and the associated (interpretable) proprioceptive features. The Gumbel network could be fed the pixel rendering of the environment, which would predict a sub-policy to apply. This sub-policy would be linear with respect to the interpretable features. We did not propose results with pixel data for now as we felt this was beyond the scope of the paper, which introduces the architecture with respect to interpretable features only. However, we do agree it would be an interesting follow-up, and intend on pursuing a pixel-based version of HC in future work.

Reviewer comment (Experiments):
The reviewer questions whether the method could be useful for debug, teach, regulate, calibrate reliance, or offer recourse, and if so, why this was not demonstrated.

Author response:
We thank the reviewer for their remark. In this work, our goal was to study the properties that could improve the interpretability of piecewise-linear policies, and to propose a parametrization that let us achieve as many of these properties as possible. Therefore, we studied and evaluated the HyperCombinator as a general-purpose architecture.

The two levels of interpretability of HC relate to its transparency. The linear coefficients inform of the importance of a feature in the computation made by a subpolicy, and the sequence of chosen behaviors gives insights about the structure of the inner decision process of the agent.
In turn, a domain expert can use this transparency to better understand the policy that they roll out (Appendix D.7 gives a low-dimensional example of such an analysis).

The sub-policy sequence visualization could also be used to detect issues at run-time, especially if no video of the agent is available. In Fig. 15 (Appendix D.6), we studied the robustness of HC to perturbations, which we could model as an adverse event. The perturbations that the agent is subjected to are translated into an unusual sub-policy sequence between timesteps 100 and 200. Hence, we could imagine a system that detects a change in the subpolicy sequence, which would in turn raise an alarm that an adverse event has been detected.

Overall, HC opens the gate to new analyses thanks to its transparency, which we think is a significant improvement from the status quo.

Reviewer comment (Evaluation):
It's not clear practically how interpretable the resulting model is. I suppose the user is supposed to inspect the linear model coefficients to understand what the policy is doing. However, there is not much discussion of this. For example, how should I interpret the coefficient heatmap in Figure 3?

Author response:
The HC architecture provides two levels of interpretability:
1. Low-level interpretability through the value of the linear coefficients of each sub-policy,
2. High-level interpretability by examining the sequence of choices of sub-policies over a trajectory.

The heatmap in Fig. 3 is an example of low-level interpretability. It shows the coefficients of each sub-policy (one column per sub-policy) for one dimension of the action space. For instance, subpolicy #0 assigns a positive coefficient to feature f0 as the corresponding cell is red. Therefore, we know that inputs with greater positive values for f0 will lead to greater predicted actions, all else being equal. Moreover, the coefficient quantifies the exact importance of feature f0 in the computation of the action.

In addition, we perform a more extensive analysis of the meaning of the coefficients in Appendix D.7, where we analyze the learnt policies on the cartpole environment. The lower dimensionality of the environment helps getting a better grasp of the computations and therefore of each sub-policy.
For instance, the coefficient values let us understand the usage of each sub-policy and how the agent specialized some to get the CartPole up, and some to maintain it straight. 
We think this is a considerable improvement over the regular non-interpretable neural network, and makes the decision making process significantly more transparent.

The high-level interpretability brought by our agent is complementary to this low-level interpretability, as we get a better understanding of how the agent processes the information to decide which subpolicy to follow. For instance, Fig. 3 (left) illustrates how the agent learnt to chain efficiently the linear sub-policies in a repeated manner to solve Half-Cheetah with a strong performance, and the hierarchical structure in Fig. 5 reflects the structure of the maze.

Reviewer comment (Evaluation):
As it stands, it seems the approach is less of a compromise between interpretability and performance and more of a deterioration of performance while only gaining a little bit of interpretability.

Author response:
While the performance of HC64 decreased with the increasing difficulty of the mazes, it never collapsed (Fig. 4). We think this is especially encouraging as these tasks are extremely challenging, and require dedicated algorithms to solve them (Chane-Sane 2021). This leads us to think that HC can handle complex environments without fully losing performance simply by increasing the number of sub-policies available. For instance, in Fig. 4, we had to trade-off some interpretability by going from 8 sub-policies to 64. 

Therefore, the answer to this question ultimately lies with the stakeholder that needs to understand the sub-policy. If that person is a domain expert, it is possible for them to increase the number of sub-policies and therefore increase the performance.
A person less familiar with the environment might want to reduce the number of sub-policies that HC can use, sacrificing some of the performance to understand how the agent interacts with the environment.
We updated the manuscript (Sec 3.2) to reflect this point (in olive).

Reviewer comment (Experiments):
How complex of environments can the HC framework handle without fully losing performance?

Author response:
While the performance of HC64 decreased with the increasing difficulty of the mazes, it never collapsed (Fig. 4). We think this is especially encouraging as these tasks are extremely challenging, and require dedicated algorithms to solve them (Chane-Sane 2021). This leads us to think that HC can handle complex environments without fully losing performance simply by increasing the number of sub-policies available. For instance, in Fig. 4, we had to trade-off some interpretability by going from 8 sub-policies to 64. 

Therefore, the answer to this question ultimately lies with the stakeholder that needs to understand the sub-policy. If that person is a domain expert, it is possible for them to increase the number of sub-policies and therefore increase the performance.
A person less familiar with the environment might want to reduce the number of sub-policies that HC can use, sacrificing some of the performance to understand how the agent interacts with the environment.
We updated the manuscript (Sec 3.2) to reflect this point (in olive).