Paper: Leveraging Hyperbolic Embeddings for Coarse-to-Fine Robot Design (ICLR-2024)

Reviewer comment (Experiments):
The generalizability of the proposed approach to other robot design representations is uncertain. Testing the approach on one common robot design representation would make it more convincing.

Author response:
**A**: Though our paper mainly focuses on multi-cellular robot design problems, our method HERD can still be extended to other robot design representations, such as articulate rigid robots. Here we provide a solution to this setting. 

To implement the idea of coarse-to-fine robot design in rigid robots, the major challenge is to define the refinement process for coarse-grained robot designs. Once the refinement definition is given, we can organize the design space as a hierarchy through recursive refinement, embed the hierarchy in hyperbolic space, and use CEM for optimization as we did in the paper. In the following, we discuss the refinement of rigid robots in detail.

Formally, a rigid robot design $D$ can be represented by a tree $D=(V,E,g,h)$, where each vertex $v\in V$ represents a joint in robot morphology and each edge $e=(v_i,v_j)\in E$ represents a limb connecting two joints $v_i,v_j$ and $|E|=|V|-1$. The function value $g(v)$ defines joint-specific attributes and function value $h(v_i,v_j)$ defines limb-specific attributes. 

The intuition behind the solution is to regard the coarse-grained robot as a simplified tree, to which we enrich details such as adding new vertices and new edges during refinement, such that the fine-grained robot has a 'similar' shape to the coarse-grained robot. Given a design $D$, we denote the finer-grained design of $D$ by $D'=(V',E',g',h')$. $D'$ is a valid refinement of $D$ if and only there is a vertex map $\pi:V'\to V$ such that the following conditions are satisfied:

(i) (coarse-to-fine condition): $|V|<|V'|$;

(ii) (edge condition): the edge $e'=(v'_i,v'_j)\in E'$ if and only if there is an edge $e=(v_1,v_2)\in E$ such that $\pi(v'_i)=v_i$ and $\pi(v'_j)=v_j$;

(iii) (vertex attribute condition): for each $v'\in V'$, we have $g'(v')=g(\pi(v'))$;

(iv) (edge attribute condition): for each $e'=(v'_i,v'_j)\in E'$, we have $h'(v_i',v_j')=h(\pi(v_i'),\pi(v_j'))$;

The above conditions enable $D'$ to be one of the refinements of $D$ with a 'similar' shape. Given this refinement definition, we can then organize the design space of rigid robots as a hierarchy like multi-cellular robots and use the hyperbolic solution for coarse-to-fine optimization. 

Due to the time limitation of rebuttal, we only provide a conceptual solution. We believe this could be a promising future work.

Reviewer comment (Theory):
The choice of Sarkar’s construction for hyperbolic embedding needs more justification.

Author response:
**A**: There are two reasons for this choice. Firstly, Sarkar's construction can embed the tree structure with arbitrarily low distortion [5], which matches our needs of embedding the tree of robot designs. Secondly, Sarkar's construction is a training-free method that will not bring extra computational cost and is also simple to implement.

Reviewer comment (Experiments):
The coarse-to-fine process seems specialized for the cellular robot representation in EvoGym. Is it generalizable to other robot design representations, such as articulate rigid robots (e.g. Wang et al. 2019 [1], Gupta 2021 [2])?

Author response:
**A**: Though our paper mainly focuses on multi-cellular robot design problems, our method HERD can still be extended to other robot design representations, such as articulate rigid robots. Here we provide a solution to this setting. 

To implement the idea of coarse-to-fine robot design in rigid robots, the major challenge is to define the refinement process for coarse-grained robot designs. Once the refinement definition is given, we can organize the design space as a hierarchy through recursive refinement, embed the hierarchy in hyperbolic space, and use CEM for optimization as we did in the paper. In the following, we discuss the refinement of rigid robots in detail.

Formally, a rigid robot design $D$ can be represented by a tree $D=(V,E,g,h)$, where each vertex $v\in V$ represents a joint in robot morphology and each edge $e=(v_i,v_j)\in E$ represents a limb connecting two joints $v_i,v_j$ and $|E|=|V|-1$. The function value $g(v)$ defines joint-specific attributes and function value $h(v_i,v_j)$ defines limb-specific attributes. 

The intuition behind the solution is to regard the coarse-grained robot as a simplified tree, to which we enrich details such as adding new vertices and new edges during refinement, such that the fine-grained robot has a 'similar' shape to the coarse-grained robot. Given a design $D$, we denote the finer-grained design of $D$ by $D'=(V',E',g',h')$. $D'$ is a valid refinement of $D$ if and only there is a vertex map $\pi:V'\to V$ such that the following conditions are satisfied:

(i) (coarse-to-fine condition): $|V|<|V'|$;

(ii) (edge condition): the edge $e'=(v'_i,v'_j)\in E'$ if and only if there is an edge $e=(v_1,v_2)\in E$ such that $\pi(v'_i)=v_i$ and $\pi(v'_j)=v_j$;

(iii) (vertex attribute condition): for each $v'\in V'$, we have $g'(v')=g(\pi(v'))$;

(iv) (edge attribute condition): for each $e'=(v'_i,v'_j)\in E'$, we have $h'(v_i',v_j')=h(\pi(v_i'),\pi(v_j'))$;

The above conditions enable $D'$ to be one of the refinements of $D$ with a 'similar' shape. Given this refinement definition, we can then organize the design space of rigid robots as a hierarchy like multi-cellular robots and use the hyperbolic solution for coarse-to-fine optimization. 

Due to the time limitation of rebuttal, we only provide a conceptual solution. We believe this could be a promising future work.

Reviewer comment (Experiments):
Since coarse-to-fine clustering is based on running K-Means with different random seeds, it seems to be possible that there are two robot designs located in different subtrees. Will such two designs be embedded in similar locations in the hyperbolic space?

Author response:
**A**: There might be some misunderstandings. For each number of clusters, we only run K-Means *once* on robot cells per experiment. In this case, two similar robot designs will always be located in the same subtrees, then they will also be embedded in similar locations in the hyperbolic space.

Reviewer comment (Evaluation):
Being relevant to the last question, it seems that the cell type is not used during the embedding. If I understand correctly, the embedding only captures the structural/subdivision similarity but not the cell functionalities. How can we interpret the distance in the embedding space into the similarity between designs?

Author response:
**A**: There might be some misunderstandings. The similarity considering cell types is captured in the robot hierarchy and the cell functionalities will also be encoded in the embeddings. Specifically, to capture their similarity, during the construction of the robot hierarchy, the child robot only changes one component compared to its parent, which leads to a high similarity of the child robots originating from the same parent robot. More generally, the similarity of any two robots in the robot hierarchy is largely determined by their distance to the nearest common ancestor robot node. If they are close to their nearest common ancestor robot node, e.g., the nearest common ancestor robot node is their shared parent robot node, they tend to be similar. This similarity is also encoded in our hyperbolic embeddings thanks to the ability of Sarkar's Construction to embed hierarchy with low distortion error.

Reviewer comment (Writing):
Page 6 “In practice, similarity here means that fine-grained robots only need to change one component to be the same as their parent robots”. What does it mean? The fine-grained robots have more clusters than the parent robot, does it mean the fine-grained robots only change the cell type of one “new” cell cluster?

Author response:
**A**: Yes, we only change the cell type of clusters of fine-grained robots. This is necessary to make child fine-grained robots similar to their parent coarse-grained robots.

Reviewer comment (Writing):
After reading the paper, the phrase “coarse-to-fine” is indeed the way of building the candidate robot design pool instead of a coarse-to-fine optimization process. If this understanding is correct, the paper (especially the abstract and introduction) needs to be more clear about this point to prevent confusion.

Author response:
**A**: There might be some misunderstandings. Our optimization is also a coarse-to-fine process but in an automatic way. Though we do not add any other mechanisms to force the robots to be finer-grained, our method gradually increases the granularity of robots solely for better performance. This phenomenon is shown in Figure 5 of our paper: as the optimization proceeds, our method samples robots from the center to the border in Poincaré ball, which is exactly the process of coarse-to-fine robot design.

Reviewer comment (Experiments):
EvoGym has 32 tasks however the algorithm is only tested on 15 of them. It would be great if the authors could evaluate the proposed algorithm on other tasks as well.

Author response:
**A**: Thanks for the constructive suggestion. We have included the results of other tasks in Figure 25 of Appendix C in the revised version of our paper. Considering all the tasks in EvoGym, HERD outperforms all the other robot design baselines (CuCo, Vanilla RL, BO, GA, CPPN-NEAT) in 15 tasks and achieves a comparable performance in 10 tasks. To our best knowledge, our work is the first one to solve this many tasks in EvoGym.

Reviewer comment (Reproducibility):
It would be great to provide the pseudo-code for the ablation variations to help understand the design of those ablation methods. For example, does HERD w/o C2F differ from full NERD mainly on the pre-sampled robot design set? So how does HERD w/o C2F sample the robot design set?

Author response:
**A**: Thanks for the constructive suggestion. We have provided the pseudo-code of the ablation variants in Algorithm 3 and 4 in Appendix B.4. Please refer to the revised version of our paper for details. 

HERD w/o C2F differs from full HERD mainly in not using the idea of coarse-to-fine. It is true that HERD w/o C2F does not use the pre-sampled robot set, instead, it maintains a distribution function that directly represents the probability of each robot cell's type. Then it uses CEM to optimize this distribution.

Reviewer comment (Theory):
From the visual results, most of the optimized designs only have one single cell type. Does this result from the limitation of the embedding approach since the embedding only captures structural information but is not informative about the cell types?

Author response:
**A**: Not really. As we discussed in Question 3, our embedding approach also captures the information about the cell types. 

This phenomenon is a proper result of optimization. As we show in Figure 5, our method has explored many robots including these single-type robots, which turned out to be the most suitable robot for the given tasks. Also please note that only coarser-grained robots tend to have one single cell type such as task PlatformJumper-v0 in Figure 3, while finer-grained robots tend to have multiple cell types such as task Carrier-v1 in Figure 3.

Reviewer comment (Reproducibility):
It's unclear to me how exactly any point on the manifold correspond to a robot configuration. Some concrete example may help understanding. If I randomly sample a point using a coordinate, what's the robot configuration it correspond to? It seems that human has to handcraft such rules.

Author response:
**A**: We would like to clarify that we do not have to handcraft such rules. Instead, we simply calculate the distance between the sampled point and other points that are embedded in the hyperbolic manifold. Then we choose the point and its corresponding robot design with minimum distance. Please refer to Figure 21 in Appendix B.2 in the revised version of our paper for a simplified visualization.

Reviewer comment (Theory):
The authors chose Poincare ball as the hyperbolic embedding space for the design tree. I am wondering how general is this? In particular, hyperbolic manifold seems very limiting. For example, let's say the robot can be comprised of material A and material B. From the center, I extend two different trees branching towards material A and material B separately. However, in each branch, fine-grained branches start to contain configurations that blends A & B which means the two branches should merge. In this case, you will have to have a finite cylinder manifold, correct?

Author response:
**A**: Thanks for the idea. In our experiment, the "branch-merge problem" might happen with a low possibility but will not influence the optimization severely. This is because our method HERD optimizes the robot designs starting from the center of the Poincaré ball, and it can automatically identify the promising sub-space regardless of whether the branches are merged or not. The idea of the finite cylinder manifold might further improve final performance, but optimizing within a three-dimensional space may also introduce additional complexities to the problem, which requires in-depth empirical investigation.

Reviewer comment (Presentation):
How exactly does the manifold correspond to designs or robot parameters? How many branches are there? I understand how do you map between the euclidean space and the hyperbolic space but this seems unclear. Giving more visualizations across the disk manifold will be helpful.

Author response:
**A**: To correspond a point in the manifold to a design, as we discussed in Weakness 1, we simply choose the pre-embedded robot design in the manifold with the minimum distance to the point.

The number of branches may vary with the level of the hierarchy of robot designs. For example, the number of branches originating from the root node only has two branches. As for the other level of the hierarchy, we restrict the number of branches originating from the same non-leaf node to $20$. This restriction may limit the search space, but since the hierarchy grows exponentially, the search space is still large enough to contain the designs that solve given tasks, as long as the number of branches exceeds a threshold.

Thanks for the suggestion of visualization. We have visualized the 'corresponding process' in a simplified disk manifold, as there are too many points in the original manifold to be visualized clearly. Please refer to Figure 21 in Appendix B.2 in the revised version of our paper for details

Reviewer comment (Experiments):
As one of the concerns of multi-cellar system, the design space is very large, so how hyperbolic solution solved this problem is not well examined.

Author response:
Our method HERD solves the large space problem of multi-cellular system by adopting a coarse-to-fine robot design mechanism, in which HERD starts by finding the optimal coarse-grained robot designs and subsequently refines them. For a simple and better optimization, we embed any-grained robots in hyperbolic space and use Cross-Entropy Method to optimize robot designs in hyperbolic space. The design space is substantially reduced in this way because (1) the design space of coarse-grained robots that HERD optimizes first is much smaller than the original design space; (2) by using the guidance of coarse-grained designs, HERD focuses on promising sub-space without the needs to explore the whole space, which further reduces the design space. This intuition is elaborated in Figure 1 and the Introduction Section of our paper. 

We have also examined the above solution empirically in our paper. We first show that HERD can successfully solve plenty of tasks in Figures 3 & 22, then we check the functionality of coarse-to-fine and hyperbolic embedding components in Figure 4, finally, we visualize the optimization process of HERD in Figure 5 for an in-depth understanding of how it works.

Reviewer comment (Evaluation):
The quantitative analysis of the proposed model is somewhat absent, some tables will help the understanding of the evaluation of different methods.

Author response:
Thanks for the suggestion. We have revised our paper and updated several tables of the performance comparisons with baselines and ablation studies in Appendix D. Specifically, we compare our method HERD with various baselines in Table 2 and HERD outperforms other robot design methods in most tasks, which showcases the effectiveness of our method. We also display the numeric results of ablation studies in Table 3 in Appendix D, which demonstrates the validity of the main components of our method: coarse-to-fine robot design and hyperbolic embedding. Please refer to the updated version of our paper for detailed quantitative results.

Reviewer comment (Evaluation):
The issue of local optimal (not very good performance compared to other baselines) and large variance: Does the parent node of an optimal robot design node consistently outperform the parent node of a non-optimal robot design node? If not, coarse-to-fine seems greedy and vulnerable to local optimal. The proposal method, HERD, appears to have no mechanism for exploration in the global design space, potentially leading to the high variance in Figure 3. And HERD is not significantly better than baselines.

Author response:
> **Weakness 1**: The issue of local optimal (not very good performance compared to other baselines). HERD is not significantly better than baselines.

**A**: We respectfully disagree with this claim. Our method HERD is significantly better than baselines as shown in Figure 3 and Figure 22 of the paper, which includes 15 tasks. In summary, (1) HERD outperforms all the other robot design baselines (CuCo, Vanilla RL, BO, GA, CPPN-NEAT) in 12 tasks out of 15 tasks; (2) HERD is much better than HandCrafted Robot in 7 tasks and has a comparable performance in 5 tasks. 

HERD utilizes a variant of the Cross-Entropy Method (CEM) for optimizing the coarse-to-fine robot design process. CEM is a popular stochastic optimization method due to its simplicity and effectiveness. Because of its population-based search and stochastic nature, it has less tendency to local optimality convergence than many other heuristic methods. Its global optimality can be further enhanced by using a mixture of Gaussian models [1], which can be an interesting direction for future work. 


> **Weakness 2**: The proposal method, HERD, appears to have no mechanism for exploration in the global design space. 

**A**: There might be some misunderstandings. HERD has an exploration mechanism, which is built inherently in the Cross-Entropy Method (CEM) -- a stochastic optimization method we use. Concretely, HERD maintains a distribution of robot designs and optimizes the distribution with CEM. When sampling from this distribution, HERD explores the robot design space such that robots of lower performance will still be sampled, though with a relatively lower possibility. 


> **Weakness 3**: The issue of high variance in Figure 3. 

**A**: We appreciate the reviewer pointing this out. High variance is a common challenge in the field of robot design. For example, GA in task Carrier-v1 and CuCo in task GapJumper-v0 also show high variance. One major cause of this problem is that the optimal control policies are possibly difficult to learn as the robot designs are changing over time. Similar robots might have totally different performances due to the learning issue of control policies. For a fair comparison, our method HERD together with baselines CuCo, Vanilla RL, and HandCrafted Robot adopt the same control policy architecture (transformers) and optimization algorithm (PPO). It is an open and important direction for future work to investigate more effective control policy learning methods in the field of robot design.

> **Weakness 4 & Question 4**: Does the parent node of an optimal robot design node consistently outperform the parent node of a non-optimal robot design node? If not, coarse-to-fine seems greedy and vulnerable to local optimal. How can we guarantee that sequential refinements on a coarse design are better than sequential refinements on another coarse design, given that the former coarse design outperforms the latter coarse design?

**A**: We do not need to guarantee that "the parent node of an optimal robot design node consistently outperforms the parent node of a non-optimal robot design node", because our method HERD is not a greedy deterministic algorithm. As we discussed in Weakness 2, HERD uses a stochastic optimization method to optimize the distribution of robot design instead of one or several robot samples. Hence, HERD can still explore child nodes from another subtree if the current child nodes are less efficient.

Reviewer comment (Experiments):
What are the policy architecture and learning methods adopted for the baselines? Are they the same as those of HERD?

Author response:
> **Question 1**: What are the policy architecture and learning methods adopted for the baselines? Are they the same as those of HERD?

**A**: The baselines CuCo, Vanilla RL, and HandCrafted Robot adopt the same control policy architecture (transformers) and learning method (PPO) as our method HERD. For the other baselines GA, BO, and CPPN-NEAT, we use their official implementations from CuCo and EvoGym.

Reviewer comment (Experiments):
HEARD w/o HE: how to integrate NGE with CEM? Could you provide more details on this ablation?

Author response:
> **Question 2**: HERD w/o HE: how to integrate NGE with CEM? Could you provide more details on this ablation?

**A**: We would like to clarify that HERD w/o HE does not integrate NGE with CEM, but combines NGE with the idea of coarse-to-fine robot design. This variant removes hyperbolic embeddings and CEM of our method HERD and uses NGE to implement coarse-to-fine robot design. We use this variant to show the effectiveness of hyperbolic embeddings. Concretely, we re-implement the original NGE in the EvoGym benchmark based on their released code. The core modification is the *mutation procedure*. To enable multi-cellular robot design, we randomly change the type of each robot cell with probability $0.1$, where the types are chosen from {Empty, Rigid, Soft, Horizontal Actuator, Vertical Actuator}. To enable coarse-to-fine robot design, we refine each robot with a probability of $0.2$ if it is coarse-grained. Then the mutated robots are viewed as offsprings and added to the robot population of NGE.

Another variant HERD w/o C2F does not adopt the idea of coarse-to-fine but uses CEM to directly search the optimal design. This ablation can verify the effectiveness of coarse-to-fine robot design. Specifically, the major difficulty of using CEM for optimizing robots in EvoGym is that the design space of EvoGym is discrete. We choose to let CEM optimize the probabilities of robot cell types ($P\in \mathbb{R}^{25\times 5}$), and take the types with maximum probabilities (`P.argmax(dim=1)`).

For a fair comparison, the control policies of all the variants in ablation studies adopt the same network architecture and optimization algorithm (PPO) as our method HERD. 

We have provided the pseudo-code of the ablation variants in Algorithm 3 and 4 in Appendix B.4. Please refer to the revised version of our paper for details.

Reviewer comment (Experiments):
How is the performance of a HandCrafted robot determined? Does it use a transformer-PPO policy? If yes, it should also be reported with variance, e.g., in Figure 3.

Author response:
> **Question 3**: How is the performance of a HandCrafted robot determined？ Does it use a transformer-PPO policy? If yes, it should also be reported with variance, e.g., in Figure 3.

**A**: The performance of the HandCrafted Robot is obtained by averaging the final performance of four runs with different random seeds. It also uses a transformer-PPO policy. The reason why we did not show its learning curve is that it is not a robot design method. HandCrafted Robot has a fixed human-designed robot morphology and only needs to learn control policy. It is unfair to directly compare this baseline with other robot design methods, whose robot morphologies are learned online. In this paper, we use the final performance improvement of our method over this baseline to show the necessity of creating robot design methods.

Nevertheless, as suggested by the reviewer, for a complete comparison, we also include the learning curve as well as its variance of HandCrafted Robot in Figure 24 of Appendix C. Our method is much better than HandCrafted Robot in 7 tasks and has comparable performance in 5 tasks. Please refer to the revised version of our paper for this new result.