Autonomy in robotic surgery is very challengingin unstructured environments, especially when interacting withdeformable soft tissues. This creates a challenge for model-based control methods that must account for deformationdynamics during tissue manipulation. Previous works in vision-based perception can capture the geometric changes withinthe scene, however, integration with dynamic properties toachieve accurate and safe model-based controllers has notbeen considered before. Considering the mechanic couplingbetween the robot and the environment, it is crucial to developa registered, simulated dynamical model. In this work, wepropose an online, continuous, real-to-simregistration methodto bridge from 3D visual perception to position-based dynamics(PBD) modeling of tissues. The PBD method is employed tosimulate soft tissue dynamics as well as rigid tool interactionsfor model-based control. Meanwhile, a vision-based strategy isused to generate 3D reconstructed point cloud surfaces that canbe used to register and update the simulation, accounting fordifferences between the simulation and the real world. To verifythis real-to-sim approach, tissue manipulation experiments havebeen conducted on the da Vinci Researach Kit. Our real-to-simapproach successfully reduced registration errors online, whichis especially important for safety during autonomous control. Moreover, the result show higher accuracy in occluded areasthan fusion-based reconstruction.
Autonomy in robotic surgery is very challenging in unstructured environments, especially when interacting with deformable soft tissues. This creates a challenge for model-based control methods that must account for deformation dynamics during tissue manipulation. Previous works in vision-based perception can capture the geometric changes within the scene, however, integration with dynamic properties toachieve accurate and safe model-based controllers has not been considered before. Considering the mechanic coupling between the robot and the environment, it is crucial to develop a registered, simulated dynamical model. In this work, we propose an online, continuous, real-to-sim registration method to bridge from 3D visual perception to position-based dynamics(PBD) modeling of tissues. The PBD method is employed to simulate soft tissue dynamics as well as rigid tool interactions for model-based control. Meanwhile, a vision-based strategy is used to generate 3D reconstructed point cloud surfaces that can be used to register and update the simulation, accounting for differences between the simulation and the real world. To verify this real-to-sim approach, tissue manipulation experiments have been conducted on the da Vinci Researach Kit. Our real-to-sim approach successfully reduced registration errors online, which is especially important for safety during autonomous control. Moreover, the result shows higher accuracy in occluded areas than fusion-based reconstruction.
R
challengingin
challenging in
R
withdeformable
with deformable
R
deformationdynamics
deformation dynamics
R
withinthe
within the
R
notbeen
not been
R
couplingbetween
coupling between
R
developa
develop a
R
wepropose
we propose
R
real-to-simregistration methodto
real-to-sim registration method to
R
tosimulate
to simulate
R
interactionsfor
interactions for
R
isused
is used
R
canbe
can be
R
fordifferences
for differences
R
verifythis
verify this
R
havebeen
have been
R
real-to-simapproach
real-to-sim approach
R
whichis
which is
R
show
shows
R
areasthan
areas than