Paper: MeMo: Meaningful, Modular Controllers Via Information Bottlenecks (ICLR-2024)

Reviewer comment (Theory):
The significance of the information bottleneck is mentioned, but a deeper exploration into its role and implications within the framework might enrich the paper's content. It might be helpful if the authors can clarify the choice of Gaussian noise as the information bottleneck and its specific advantages over other potential techniques. Providing some theoretical underpinnings would enhance the paper's contribution.

Author response:
While we agree that our work would benefit from greater theoretical analysis on the effect of the bottleneck, we believe that there is an intuitive explanation behind the Gaussian noise bottleneck, which we detail in our global response (Part 1). As a result of the added noise, the modules treat the master signal not as a high-dimensional vector but instead as a much lower-dimensional representation, where many of the dimensions have redundant information. Our eigenvalue experiments in Section 4.4 demonstrate that the modules do learn low dimensional representations of the actuator space with respect to the master controller, meaning that the modules themselves must be expressive enough to capture the variance in the actuator space that is not accounted for by the master signal.

Reviewer comment (Experiments):
While the paper mentions comparisons with other methods like NerveNet, a more exhaustive comparative analysis might accentuate MeMo's advantages more distinctly. It would be beneficial to provide more granularity on experimental setups, parameter settings, and perhaps even failed experiments to give readers a comprehensive understanding. Please consider extending experiments to more diverse robot morphologies or even real-world robot applications to further validate the framework's robustness.

Author response:
In addition to the hyperparameters detailed in Appendix A.2, we have performed additional experiments increasing the number of layers of the master controller and the modules, showing that our framework achieves similar results with deeper MLPs on the 6 to 12 leg centipede transfer. We note that for our framework, we can use most of the same hyperparameters for PPO used to train regular 2 layer MLPs on the same task. We additionally find it sufficient to try smaller batch sizes at transfer time, and we have detailed the specific batch sizes that yield these improvements in Appendix A.2.

Reviewer comment (Novelty):
This article is not very innovative to the field.

Author response:
While many papers [1, 2, 3] have studied the problem of learning modular controllers, these methods take the approach of training on many environments that represent a diverse set of morphologies. However, we focus on an orthogonal dimension of the problem, where we assume that the expert only has a single training robot and wants a fast turnaround when training a new controller for a modified version of the robot, rather than spending significant compute and time to perform multi-task RL. To the best of our knowledge, only NerveNet attempts to do one-shot structure transfer. 

[1] Huang, Wenlong et al. “One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control.” 
[2] Kurin, Vitaly et al. "My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control."
[3] Gupta et al. "MetaMorph: Learning Universal Controllers with Transformers."

Reviewer comment (Experiments):
I have concerns on extending this method to more complex tasks (environments). The authors should compare their methods with more strong baselines which learn universal controllers for modular robots.

Author response:
The focus of our method is structure transfer, and we see task transfer as an additional benefit on top of the structure transfer we achieve. We agree that because our approach is not specifically designed to tackle task transfer, we only expect for task transfer to be successful when the source and transfer tasks are similar enough. We show the task transfer results as supporting evidence to our structure transfer results that our method does learn meaningful modules. In particular, on task transfer, our method significantly outperforms the hierarchical architecture pretrained end-to-end with RL, which highlights the benefit of the training method we propose.

Reviewer comment (Experiments):
Can and how MeMo transfer to the real world problem?

Author response:
We believe that extending our work to real world problems is a promising future direction and that our experiments have shown evidence that our method has potential to work in real-world applications. Across our examples we have tried a diversity of module shapes that would be commonly found in rigid robots for locomotion and manipulation. As mentioned in our response to reviewer aQSN, visualizations of learned policy’s trajectories show that the modules can have complex coordination mechanics, particularly the behaviors of each leg of the 12 leg centipede.

Reviewer comment (Experiments):
The empirical results can be further improved. The authors only compared the proposed method with NerveNet, but ignored any recent works in the field of modular RL, such as SMP (One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control), Amorpheus (My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control), METAMORPH: LEARNING UNIVERSAL CONTROLLERS WITH TRANSFORMERS, etc.

Author response:
Please see our global response (Part 2) for a complete discussion of how our work is related to the literature on learning modular controllers via multi task reinforcement learning. In particular, works that leverage Transformer-based policies (e.g. Amorpheus, MetaMorph) primarily rely on node features, since Transformers disregard connectivity between limbs [1]. As such, they depend on a multi-task setting to generalize, making them fundamentally unsuited to our problem. 

[1] Kurin, Vitaly et al. "My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control."

Reviewer comment (Theory):
The proposed method requires predefining the partition of robots.

Author response:
We understand that our method’s requirement for a predefined partitioning of joints can be viewed as a limitation. However, we believe that in many cases, it is desirable for experts to inject domain knowledge in our algorithm via their partitioning, which can account for physical constraints. In addition, because our framework only requires the partitioning of joints for a single robot, we do not believe our approach is significantly limited by requirements for expert labels.

Reviewer comment (Evaluation):
This paper does not provide any discussions of limitations.

Author response:
Please see our updated paper for a full discussion of the limitations.

Reviewer comment (Reproducibility):
How does the MLP network in Figure 4 handle incompatible input and output?

Author response:
$M_1$, $M_2$, and $M_3 $ are all different MLP’s to avoid the incompatibility of the observation spaces seen by $M_2$ and $M_3$. While this requires $M_3$ to be retrained during structure transfer, our experiments show that such retraining is highly sample efficient.

Reviewer comment (Theory):
Why does adding Gaussian noise can be regarded as an information bottleneck?

Author response:
For a detailed explanation, please see our global response (Part 1). In Section 4.4, with our eigenvalue experiments, we have shown that the modules learn low dimensional representations of the actuator space with respect to the master controller. This means that the modules must be expressive enough to capture the variance in the actuator space that is abstracted away by the master signal.

Reviewer comment (Theory):
The proposed method was not sufficiently / rigorously analyzed. It is not clear why certain choices were made: the Gaussian-noise induced 'bottleneck' for instance performed better than L1-regularization, but it would be nice to analyze this more mathematically and study how it induces modularity during training.

Author response:
Please see our global response (Part 1) for a detailed explanation of the intuition behind the Gaussian noise bottleneck. While we agree that our work would benefit from greater theoretical analysis on the effect of the bottleneck, we believe that our results have an intuitive explanation. When noise is added to the master signal, the modules treat the signal as a low-dimensional representation with redundancy. Our eigenvalue experiments empirically demonstrate that the interface between the master and the modules is low-dimensional, forcing the modules to capture the variance in the actuator space that the master controller’s signal abstracts away.

Reviewer comment (Experiments):
It is very difficult to convincingly show knowledge/policy transfer between different 'robots', when these are only in simulation. The robots used in simulation, moreover, have very similar structures, for a more convincing argument I would show more experiments between quite different morphologies and tasks. Even without real robot experiments, it would go a long way to consider several difficulties that show up in real robot experiments: sensor noise, not having access to joint velocity measurements, actuator limits etc.

Author response:
As discussed in our global response (Part 1), transfer to diverse morphologies is outside the scope of our work and have revised our Introduction and added a new section on limitations and future work to clarify this point. We note that even in simulation, visualizations of learned policy’s trajectories show that the modules can have complex behavior, particularly the behaviors of each leg of the 12 leg centipede. However, we agree that extending our work to real world robots is a direction for future work.

Reviewer comment (Evaluation):
It would be nice to show when and if the proposed method fails: this would help to understand how the method contributes to the literature better and also pave the way to more research.

Author response:
We have updated our paper with an additional section on limitations and directions for future work.

Reviewer comment (Experiments):
Shallow MLPs were used throughout the training as components, have the authors tried other architectures such as convolutional networks, transformers, or deeper MLPs?

Author response:
We have run additional experiments increasing the master controller and module size from 2 to 4 layers MLPs (Fig. 12). On the 6 to 12 leg centipede transfer, these variants, when trained with our framework, perform comparably to the original architecture.

Reviewer comment (Experiments):
Are there other approaches to compare against besides Nerve-Net? The related work sections mentions many recent papers, some of which could be comparable.

Author response:
Much of the related work explores a fundamentally different problem from our work, of training universal controllers, typically Transformers, over multi-task datasets. As Transformers’ capability to perform tasks rely on attention over node observations, they depend on a multi-task setting to generalize effectively, whereas our single shot transfer necessitates leveraging the modular structure of a single robot as much as possible. Please see our global response (Part 2) for a complete discussion of how our work is related to the literature on learning modular controllers via multi task reinforcement learning.