Paper: Autonomous Catheterization with Open-source Simulator and Expert Trajectory (ICLR-2024)

Reviewer comment (Novelty):
ENN model architecture is not state-of-the-art, e.g. no transformer.

Author response:
We acknowledge the absence of a transformer in the ENN model architecture. Our initial trials included experimenting with a transformer architecture, specifically based on the visual transformer framework [^1]. However, this approach did not yield the desired performance. In our experiments, the transformer-based model required approximately 4.5 hours of training but failed to successfully complete the task within the allocated time frame. Therefore, we opted for the current ENN model architecture, which demonstrated superior performance and efficiency in achieving the task objectives. We now include these details in *Appendix J*.

[^1]: Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S. and Uszkoreit, J., 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.

Reviewer comment (Reproducibility):
Lack of relevant details, e.g. what is the expert policy that the ENN is trained with. Why don’t you show results for the expert policy in the table?

Author response:
To clarify, the ENN in our study is a standalone system, representing a multi-modal network combined with a soft actor-critic approach. It does not utilize or train with a separate *expert policy*, hence there are no additional policy results to present in the table.

Reviewer comment (Novelty):
If an expert policy exists, why do you train an ML model (ENN) model for the expert navigation task?

Author response:
As per *Comment 3*, the policy is part of the network.

Reviewer comment (Presentation):
Some figures / tables are not easy to understand on their own, e.g. Table 2, for which e.g. instead of Q1 you could write “Anatomical accuracy”.

Author response:
Thank you for your feedback on the clarity of our figures and tables. As per your suggestion, we explicitly state the assessed properties in *Table 2* and rewrote the description to match within *Section 5.1*, under the *User Study* paragraph.

| Question | Average | STD |
|----------|---------|-----|
| Anatomical Accuracy | 4\.57 | 0\.53 |
| Navigation Realism | 3\.86 | 0\.69 |
| User Satisfaction | 4\.43 | 0\.53 |
| Friction Accuracy | 4\.00 | 0\.82 |
| Interaction Realism | 3\.75 | 0\.96 |
| Motion Accuracy | 4\.25 | 0\.50 |
| Visual Realism | 3\.67 | 1\.15 |

Reviewer comment (Novelty):
The implications for the ML community were not clearly described. Authors should discuss the importance of generating simulated images, the relationship to other generative methods (e.g., purely DL-based approaches that may not capture underlying physics), and most importantly, why the approach is important to ML.

Author response:
We want to emphasize that the rich diversity of the simulated data addresses the challenge of data scarcity, enhancing the robustness and generalization ability of ML models. Moreover, our method opens up new possibilities for ML research in specialized medical fields, previously constrained due to limited data availability, thus making a significant contribution to advancing ML applications in critical healthcare domains. We highlight this in the *Introduction*.

Reviewer comment (Writing):
It is not clear what is a guidewire and what is its purpose.

Author response:
We have included a description in *Section 3, The CathSim Simulator* within the *Guidewire* paragraph.

> ***Guidewire.**** A rope-like structure designed to direct the catheter towards its intended position. The guidewire is composed of the main body and a tip, where the tip is characterized by a lower stiffness and a specific shape (depending on the procedure)*

Reviewer comment (Reproducibility):
It is not clear how force labels are extracted from CatSim outputs, which seem to be images + joint position info (as discussed in Section 4.1).

Author response:
Given that we have access to the simulation environment, we extract the normal and frictional forces and compute the Euclidean norm. These are then averaged across all contacts. This has been discussed in *Appendix, Section C* under the *Force* paragraph.

Reviewer comment (Writing):
In Section 4.1, authors mention that the output is a feature vector Z, but it is unclear how it is used afterwards.

Author response:
We clarify that in our implementation, the feature vector ($Z$) is utilized as the input for the soft actor-critic policy. This allows the policy ($\pi$) to approximate the optimal action based on the given feature vector ($Z$). We clarify this in *Section 4.1 Expert Navigation Network* under the second paragraph.

> *The feature vector (Z) serves as the input for training the soft-actor-critic (SAC) policy ($\pi$), a core component of our reinforcement learning approach.*

Reviewer comment (Experiments):
More information about the number of labelled training samples would be helpful (e.g., in Section 4, where authors state “vast amount of labeled training samples” but do not provide additional details).

Author response:
The details contained in the experimental section, namely within *Section 5.2*, the *Expert Trajectory vs. Human Skill* paragraph, where we state that a number of $10^5$ time steps are sampled.

> *As depicted in Fig. 7, within the BCA, our algorithms successfully navigate the environment after $10^5$ time steps and half of them exhibited superior performance compared to the human operator.*

Reviewer comment (Writing):
Minor point: there is a typo in contribution 1: “and AR/VR ready” -> “and is AR/VR ready”.

Author response:
Thank you for pointing out. The phrase “and AR/VR ready” in Contribution 1 is intentionally phrased as such for brevity and stylistic choice.

Reviewer comment (Writing):
BCA and LCCA are not defined.

Author response:
We defined the brachiocephalic artery (BCA) and the left common carotid artery (LCCA) in our manuscript. The initial definitions were included in *Figure 8*. Recognizing the need for clarity, we have now *additionally* incorporated these definitions into the main text. In *Section 5.2*, under the *Experimental Setup* paragraph, we now explicitly state:

> "*Two targets are selected for the procedures, specifically the brachiocephalic artery (BCA) and the left common carotid artery (LCCA).*"

Reviewer comment (Evaluation):
It would help if the authors split user study questions by type (realism and satisfaction), so it is easier to interpret results.

Author response:
**Categorization of Questions**

We now explicitly name the questions based on the assessed property:

| Question | Average | STD |
|----------|---------|-----|
| Anatomical Accuracy | 4\.57 | 0\.53 |
| Navigation Realism | 3\.86 | 0\.69 |
| User Satisfaction | 4\.43 | 0\.53 |
| Friction Accuracy | 4\.00 | 0\.82 |
| Interaction Realism | 3\.75 | 0\.96 |
| Motion Accuracy | 4\.25 | 0\.50 |
| Visual Realism | 3\.67 | 1\.15 |

Reviewer comment (Presentation):
Table 4 and 5 include various types of inputs (Image, Mask, Internal, Human). A visualization of what these inputs are would help in understanding what is being compared.

Author response:
**Representation of Inputs**

The inputs mentioned in Tables 4 and 5, including Image, Mask, Internal, and Human types, are illustrated in *Figure 4* under *Section 4*, providing a clear visualization for visual understanding.

Reviewer comment (Theory):
If the goal is to create an autonomous catheterization technique, why is integration of expert trajectories important and/or beneficial? It would seem the opposite is true.

Author response:
IL is used to demonstrate the downstream tasks that our simulator can be used for. In this task, we use the expert to aid the training of an IL agent. This has explicit value in sim-to-real transfer. We emphasize this in *subsection 4.2*, particularly in the *Imitation Learning* paragraph.

The use of Imitation Learning (IL) is crucial in training an IL agent for autonomous catheterization. Expert trajectories provide a foundational dataset that enables the IL agent to learn efficient, safe, and effective maneuvers. This methodology is vital for sim-to-real transfer, ensuring that the autonomous system performs reliably in real-world scenarios. Detailed evidence supporting this approach is presented in *Subsection 4.2*, particularly in the *Imitation Learning paragraph*, where we demonstrate improved performance metrics including lower force, shorter path lengths, and higher safety and success rates.

> ***Imitation Learning.**** We utilize our ENN using behavioral cloning, a form of imitation learning, to train our navigation algorithm in a supervised manner. This approach emphasizes the utility of the simulation environment in extracting
> meaningful representations for imitation learning purposes. Firstly, we generate expert trajectories by executing the expert policy, denoted as $\\pi\_{\\rm exp}$, within CathSim. These trajectories serve as our labeled training data, providing the desired actions for each state encountered during
> navigation. Secondly, to mimic the real-world observation space, we select the image as the only input modality. Thirdly, we train the behavioral cloning algorithm by learning to replicate the expert's actions given the input observations and optimizing the policy parameters to minimize the
> discrepancy between the expert actions and the predicted actions:*

Reviewer comment (Writing):
I was very confused about the presentation. For example, I didn't find the definition of BCA, LCCA.

Author response:
We defined the brachiocephalic artery (BCA) and the left common carotid artery (LCCA) in our manuscript. The initial definitions were included in *Figure 8*. Recognizing the need for clarity, we have now *additionally* incorporated these definitions into the main text. In *Section 5.2*, under the *Experimental Setup* paragraph, we now explicitly state:

> "*Two targets are selected for the procedures, specifically the brachiocephalic artery (BCA) and the left common carotid artery (LCCA).*"

Reviewer comment (Experiments):
Since this is a simulator designed specifically for endovascular robots, it will make little sense if the application of it involves modalities that is very inaccessible: for example the image as shown in the first input to the expert navigation network. I would expect real-world images are highly diverse than rendering images that has clean background, unified rendering parameters, and a fixed camera position. I don't think the propose ENN really showed the importance of the work.

Author response:
The images generated by our simulator can be adapted for realistic X-Ray imaging scenarios. This adaptation is achieved using advanced domain adaptation techniques, further elaborated in our referenced work[^1]. Furthermore, initial real-world tests, conducted on physical aortic arch models, have produced results closely mirroring our simulated outcomes[^2]. This underscores the practical relevance and validity of our simulation for endovascular robotic applications.

[^1]: Kang, J., Jianu, T., Huang, B., Bhattarai, B., Le, N., Coenen, F. and Nguyen, A., 2023. Translating Simulation Images to X-ray Images via Multi-Scale Semantic Matching. arXiv preprint arXiv:2304.07693.

[^2]: Kundrat, D., Dagnino, G., Kwok, T.M., Abdelaziz, M.E., Chi, W., Nguyen, A., Riga, C. and Yang, G.Z., 2021. An MR-Safe endovascular robotic platform: Design, control, and ex-vivo evaluation. IEEE transactions on biomedical engineering, 68(10), pp.3110-3121.

Reviewer comment (Experiments):
I am not sure why the authors skipped AR/VR and blood simulation parts in the experiment section but still claim the contribution of them in the table. I have a feeling that these contributions are not grounded.

Author response:
Our manuscript primarily discusses experiments focusing on core simulation aspects. The AR/VR and blood simulation components, while crucial, were not the experiment's main focus as are not directly related to ML. These are, however, fully implemented and accessible in our GitHub repository.

**The blood simulation** is detailed under the `fluid-branch` of our repo. The blood flow simulation was executed using ANSYS, tailored to a silicone-based anthropomorphic phantom from Elastrat Sarl Ltd., Switzerland. The simulation's output is formatted into spatial $(x, y, z)$ and corresponding velocity $(v_x, v_y, v_z)$ components. This data structure allows us to implement a query function that retrieves specific velocity vectors based on the input spatial coordinates. This approach precisely models blood movement within the phantom, whilst being computationally efficient.

Reviewer comment (Novelty):
My main reservation is the contribution. The manuscript mainly created a new environment based on MuJoCo and only showed the effectiveness of the simulator in simulated tasks with rendered inputs. I need more evidence why the new environment based on an existing simulator is valuable to the community.

Author response:
Our simulator offers significant value to the community due to its unique features and capabilities:

- **Rapid ML Algorithm Development:** Provides an *off-the-shelf* solution for developing machine learning algorithms for endovascular navigation, featuring easy installation and gymnasium support.
- **Anatomically Accurate Phantoms:** Includes realistic, high-fidelity aortic models from Elastrat Sarl Ltd., Switzerland, enhancing the simulator's practical relevance for medical research.
- **Diverse Aortic Models:** Incorporates four distinct aortic arch models (Type-I, Type-II, Type-I with aneurysm, and low tortuosity based on patient-specific CT scan), broadening the range of anatomical structures for comprehensive simulation. The aortic models can be visualized in *Appendix A, Figure 11*
- **High Training Speed:** Achieves a balance between computational demand and efficiency, supporting 40 to 80 frames per second performance across various algorithms, as shown in our FPS comparison (*Appendix B, Figure 12*).
- **Advanced Aorta Modeling:** Utilizes detailed 3D mesh representations and efficient collision modeling techniques for realistic and computationally effective aorta simulation.
- **Realistic Guidewire Simulation:** Models the guidewire's flexibility and motion accurately, using a low-cost computational approach that mimics real catheter behavior.
- **Compatibility with AR/VR Training:** Integrates seamlessly into Unity, enabling advanced AR/VR surgeon training applications. This can be seen on [GitHub](https://github.com/airvlab/cathsim/tree/fluid)
- **Targeted Algorithm Development:** Facilitates the creation of algorithms tailored to specific aortic complications, contributing to more focused and effective medical interventions. We show this through the downstream tasks employed.

This combination of features makes our simulator not only an extension of MuJoCo but a valuable, versatile tool for both the machine learning and medical communities.