  Real-time surgical phase recognition is a fundamental task in modern
operating rooms. Previous works tackle this task relying on architectures
arranged in spatio-temporal order, however, the supportive benefits of
intermediate spatial features are not considered. In this paper, we introduce,
for the first time in surgical workflow analysis, Transformer to reconsider the
ignored complementary effects of spatial and temporal features for accurate
surgical phase recognition. Our hybrid embedding aggregation Transformer fuses
cleverly designed spatial and temporal embeddings by allowing for active
queries based on spatial information from temporal embedding sequences. More
importantly, our framework is lightweight and processes the hybrid embeddings
in parallel to achieve a high inference speed. Our method is thoroughly
validated on two large surgical video datasets, i.e., Cholec80 and M2CAI16
Challenge datasets, and significantly outperforms the state-of-the-art
approaches at a processing speed of 91 fps.
