  Human activities can be learned from video. With effective modeling it is
possible to discover not only the action labels but also the temporal
structures of the activities such as the progression of the sub-activities.
Automatically recognizing such structure from raw video signal is a new
capability that promises authentic modeling and successful recognition of
human-object interactions. Toward this goal, we introduce Asynchronous-Sparse
Interaction Graph Networks (ASSIGN), a recurrent graph network that is able to
automatically detect the structure of interaction events associated with
entities in a video scene. ASSIGN pioneers learning of autonomous behavior of
video entities including their dynamic structure and their interaction with the
coexisting neighbors. Entities' lives in our model are asynchronous to those of
others therefore more flexible in adaptation to complex scenarios. Their
interactions are sparse in time hence more faithful to the true underlying
nature and more robust in inference and learning. ASSIGN is tested on
human-object interaction recognition and shows superior performance in
segmenting and labeling of human sub-activities and object affordances from raw
videos. The native ability for discovering temporal structures of the model
also eliminates the dependence on external segmentation that was previously
mandatory for this task.
