  Automated surgical gesture recognition is of great importance in
robot-assisted minimally invasive surgery. However, existing methods assume
that training and testing data are from the same domain, which suffers from
severe performance degradation when a domain gap exists, such as the simulator
and real robot. In this paper, we propose a novel unsupervised domain
adaptation framework which can simultaneously transfer multi-modality
knowledge, i.e., both kinematic and visual data, from simulator to real robot.
It remedies the domain gap with enhanced transferable features by using
temporal cues in videos, and inherent correlations in multi-modal towards
recognizing gesture. Specifically, we first propose an MDO-K to align
kinematics, which exploits temporal continuity to transfer motion directions
with smaller gap rather than position values, relieving the adaptation burden.
Moreover, we propose a KV-Relation-ATT to transfer the co-occurrence signals of
kinematics and vision. Such features attended by correlation similarity are
more informative for enhancing domain-invariance of the model. Two feature
alignment strategies benefit the model mutually during the end-to-end learning
process. We extensively evaluate our method for gesture recognition using DESK
dataset with peg transfer procedure. Results show that our approach recovers
the performance with great improvement gains, up to 12.91% in ACC and 20.16% in
F1score without using any annotations in real robot.
