Paper: Generation of Geodesics with Actor-Critic Reinforcement Learning to Predict Midpoints (ICLR-2024)

Reviewer comment (Experiments):
The empirical evaluation environments are relatively artificial. I would expect some more practical tasks such as robotic motion planning to be more effective in demonstrating the significance of the contribution.

Author response:
The first concern that the experimental environments were artificial could have been answered by conducting an additional experiment.

Reviewer comment (Experiments):
The current baselines are all RL based. I think some classical motion planning algorithms should be included too, such as RRT (RRT*) and A* search.

Author response:
One of the reasons we focused the baselines on reinforcement learning is that it is difficult to make a fair comparison with searching methods.
While searching methods can find good solutions if enough time is spent, prediction by trained policy uses very little time.
Therefore, we decided to limit ourselves to RL methods for situations where we want to generate paths or motions in a short time.

Reviewer comment (Writing):
This is more of a clarification of the problem setting. It seems that the end goal of the learned policy is not necessarily finding the shortest path. The success criterion is stated as “all values of C(4) for two consecutive points are not greater than $\epsilon$”, which does not imply that a path is the shortest. Is this correct? If so, this should be stated more clearly.

Author response:
Indeed, judged as a success does not mean being the shortest, but success rates should increase if shorter paths are chosen for all pairs. The reason we do not use the lengths of paths for evaluation is that, since the metric can only be calculated locally, lengths cannot be calculated when the success condition is satisfied. Moreover, it is needed to compare fairly with the sequential baseline method, which has a different generation method. For these reasons, we chose this evaluation scheme.

Reviewer comment (Theory):
Please provide some motivations for the definition of $C$ (Equation (4)). Also please explain what $df_x$ is in this definition.

Author response:
Sorry for the missing explanation. $df_x$ is the differential (pushfoward) of $f$ at $x$.

Since the Finsler distance is defined by (1) and the geodesics can be approximated by the segment in the coordinate space,
we approximate the distance between the closed points by (5). Furthermore, since $F$ is continuous and the points are closed, this integral can be approximated by the value at $t=0$. This is (4).

Reviewer comment (Theory):
In Equation (11), should the right-hand side be $d(x, y)$, the true distance rather than the local approximation? Either way, Equation (11) could use a more expanded explanation.

Author response:
Unlike the proposed method, different policies and value functions are considered here for different depths.
$V_i$ represents the sums of the values of $C$ after the policies $\pi_{i-1}, \ldots ,\pi_0$ is applied.
It is therefore correct that the right-hand side of (11) is $C$.

Reviewer comment (Writing):
In Proposition 2, what is $V_i$?

Author response:
This is a typo of $V_i^*$. Thank you for pointing this out.

Reviewer comment (Experiments):
In the Sequential Reinforcement Learning (Seq) baseline, why is the reward function (Equation (16)) scaled by $\epsilon$? How does this decision affect the learning?

Author response:
This motivates agents to reach goals in as few steps as possible.

Reviewer comment (Experiments):
The paper lacks a broader range of examples to demonstrate the applicability of the method to more common robotic tasks like locomotion and manipulation planning.

Author response:
> Can the authors provide additional examples where their method might be applicable, specifically within the realm of robotics tasks like locomotion and manipulation?

Some robotics tasks including manipulation are formulated by Riemannian geometry in the context of Riemannian Motion Optimization.
See the following papers:

Ratliff, Nathan, Marc Toussaint, and Stefan Schaal. "Understanding the geometry of workspace obstacles in motion optimization." 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2015.

Mainprice, Jim, Nathan Ratliff, and Stefan Schaal. "Warping the workspace geometry with electric potentials for motion optimization of manipulation tasks." 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2016.

Therefore, our method is potentially appliable to these tasks.
Especially, since the technique to warp metrics for avoiding obstacles is already studied, our method can be appliable to environments with obstacles.

Reviewer comment (Theory):
The algorithm requires additional assumptions that may not be readily available or applicable in common robotic tasks, such as locomotion and manipulation planning. These assumptions include the need for global coordinate systems, obstacle-free environments, and environment-specific policy learning. The method's effectiveness is contingent on these conditions, which are not always present in more complex or dynamically changing real-world scenarios. Additionally, the challenge of generating globally optimal paths and dealing with the complexity of Finsler geodesics further limits its applicability to standard reinforcement learning tasks.

Author response:
The sentence "only works well locally since we assume that manifolds have global coordinate systems and the continuous midpoint property may be satisfied only locally" we wrote is maybe misleading. We assumed the continuous midpoint property for the theoretical result, it seems not to be practically necessary. Indeed, even environments we used for experiments seem not to have the property.
Therefore, this assumption is a limitation of our theoretical contribution and not of our method.

Reviewer comment (Theory):
The paper states that the method "only works well locally since we assume that manifolds have global coordinate systems and the continuous midpoint property may be satisfied only locally. For the generation of globally minimizing geodesics, we may have to divide manifolds, train policies for each local region and connect locally generated geodesics." It also states that "the policy has to be learned for each environment. By modifying our method so that the actor and critic input information on environments, it may be possible to learn a policy applicable to different environments." These statements highlight the limitations regarding the need for specific geometric and topological assumptions that may not hold in typical RL tasks in robotics.

Author response:
The sentence "only works well locally since we assume that manifolds have global coordinate systems and the continuous midpoint property may be satisfied only locally" we wrote is maybe misleading. We assumed the continuous midpoint property for the theoretical result, it seems not to be practically necessary. Indeed, even environments we used for experiments seem not to have the property.
Therefore, this assumption is a limitation of our theoretical contribution and not of our method.

Reviewer comment (Novelty):
A line of work on quasimetric distance for goal-conditioned RL seems related, which could provide important context and benchmarking. It is unclear whether the proposed approach is related to them. Relevant works include:  
- Tongzhou Wang et al., Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning, ICML 2023  
- Tongzhou Wang et al., On the Learning and Learnability of Quasimetrics, ICLR 2022

Author response:
> Could the authors discuss the relationship and distinctions between their work and recent research on quasimetric learning for goal-conditioned RL?

Thank you for sharing related works!
The main difference between our baseline sequential RL and the quasimetric learning method is architectures of value function approximation models. 
Since we do not focus on architectures of critics and used simple MLPs for both the proposed method and the baseline methods, the direction of our research and the direction of this research are orthogonal. The embedding method for quasimetric learning can be used for the critic in our method.