  Automatic surgical workflow recognition is a key component for developing
context-aware computer-assisted systems in the operating theatre. Previous
works either jointly modeled the spatial features with short fixed-range
temporal information, or separately learned visual and long temporal cues. In
this paper, we propose a novel end-to-end temporal memory relation network
(TMRNet) for relating long-range and multi-scale temporal patterns to augment
the present features. We establish a long-range memory bank to serve as a
memory cell storing the rich supportive information. Through our designed
temporal variation layer, the supportive cues are further enhanced by
multi-scale temporal-only convolutions. To effectively incorporate the two
types of cues without disturbing the joint learning of spatio-temporal
features, we introduce a non-local bank operator to attentively relate the past
to the present. In this regard, our TMRNet enables the current feature to view
the long-range temporal dependency, as well as tolerate complex temporal
extents. We have extensively validated our approach on two benchmark surgical
video datasets, M2CAI challenge dataset and Cholec80 dataset. Experimental
results demonstrate the outstanding performance of our method, consistently
exceeding the state-of-the-art methods by a large margin (e.g., 67.0% v.s.
78.9% Jaccard on Cholec80 dataset).
