  Long-term complex activity recognition and localisation can be crucial for
the decision-making process of several autonomous systems, such as smart cars
and surgical robots. Nonetheless, most current methods are designed to merely
localise short-term action/activities or combinations of atomic actions that
only last for a few frames or seconds. In this paper, we address the problem of
long-term complex activity detection via a novel deformable, spatiotemporal
parts-based model. Our framework consists of three main building blocks: (i)
action tube detection, (ii) the modelling of the deformable geometry of parts,
and (iii) a sparsity mechanism. Firstly, action tubes are detected in a series
of snippets using an action tube detector. Next, a new 3D deformable RoI
pooling layer is designed for learning the flexible, deformable geometry of the
constellation of parts. Finally, a sparsity strategy differentiates between
activated and deactivate features. We also provide temporal complex activity
annotation for the recently released ROAD autonomous driving dataset and the
SARAS-ESAD surgical action dataset, to validate our method and show the
adaptability of our framework to different domains. As they both contain long
videos portraying long-term activities they can be used as benchmarks for
future work in this area.
